[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:50:04 dist-test-slave-bndk krb5kdc[3965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:50:04 dist-test-slave-bndk krb5kdc[3965](info): set up 2 sockets
Aug 11 11:50:04 dist-test-slave-bndk krb5kdc[3965](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260811 11:50:05.356161 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.170s user 0.010s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:50:05 dist-test-slave-bndk krb5kdc[3965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:50:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:50:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:50:05.495255 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:36153
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:46701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:36153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:05.628638 3989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:05.628944 3989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:05.628988 3989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:05.633131 3989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:50:05.633246 3989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:05.633278 3989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:05.633299 3989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:50:05.633337 3989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:50:05.639168 3989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:46701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:36153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:36153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:05.640772 3989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:05.642155 3989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:50:05.656208 3989 server_base.cc:1034] running on GCE node
W20260811 11:50:05.656243 3997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:05.656363 3998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:05.656901 4001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:05.657842 3989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:05.659854 3989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:05.661055 3989 hybrid_clock.cc:648] HybridClock initialized: now 1786449005661011 us; error 70 us; skew 500 ppm
Aug 11 11:50:05 dist-test-slave-bndk krb5kdc[3965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449005, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:05.664242 3989 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:50:05.665644 3989 webserver.cc:460] Webserver started at http://127.3.221.126:34937/ using document root <none> and password file <none>
I20260811 11:50:05.666694 3989 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:05.666779 3989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:05.667203 3989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:05.670151 3989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "51f9f6ca7ad24c0485f8e41b933675b4"
format_stamp: "Formatted at 2026-08-11 11:50:05 on dist-test-slave-bndk"
server_key: "6a26c32f5226ba5a8b3b44848c94b058"
server_key_iv: "ca2baeeb9937517a423d33597e2379f8"
server_key_version: "encryptionkey@0"
I20260811 11:50:05.670742 3989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "51f9f6ca7ad24c0485f8e41b933675b4"
format_stamp: "Formatted at 2026-08-11 11:50:05 on dist-test-slave-bndk"
server_key: "6a26c32f5226ba5a8b3b44848c94b058"
server_key_iv: "ca2baeeb9937517a423d33597e2379f8"
server_key_version: "encryptionkey@0"
I20260811 11:50:05.676380 3989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 11:50:05.680402 4011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:05.682284 3989 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 11:50:05.682518 3989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "51f9f6ca7ad24c0485f8e41b933675b4"
format_stamp: "Formatted at 2026-08-11 11:50:05 on dist-test-slave-bndk"
server_key: "6a26c32f5226ba5a8b3b44848c94b058"
server_key_iv: "ca2baeeb9937517a423d33597e2379f8"
server_key_version: "encryptionkey@0"
I20260811 11:50:05.682646 3989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:05.706704 3989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:05.710690 3989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:05.710933 3989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:05.721325 3989 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:36153
I20260811 11:50:05.721362 4114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:36153 every 8 connection(s)
I20260811 11:50:05.722901 3989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:50:05.725617 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 3989
I20260811 11:50:05.725783 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:50:05.727330 3957 external_mini_cluster.cc:1428] Setting key 400ce905780c9070a1116eaea6be9a72
I20260811 11:50:05.728523 4115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:05.739886 4115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4: Bootstrap starting.
I20260811 11:50:05.745124 4115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:05.746733 4115 log.cc:824] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:05.750280 4115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4: No bootstrap required, opened a new log
Aug 11 11:50:05 dist-test-slave-bndk krb5kdc[3965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:05.755813 4115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51f9f6ca7ad24c0485f8e41b933675b4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36153 } }
I20260811 11:50:05.756187 4115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:05.756297 4115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51f9f6ca7ad24c0485f8e41b933675b4, State: Initialized, Role: FOLLOWER
I20260811 11:50:05.757205 4115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51f9f6ca7ad24c0485f8e41b933675b4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36153 } }
I20260811 11:50:05.757506 4115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:50:05.757663 4115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:50:05.757835 4115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:05.759074 4115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51f9f6ca7ad24c0485f8e41b933675b4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36153 } }
I20260811 11:50:05.759274 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:05.734550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39510 (local address 127.3.221.126:36153)
0811 11:50:05.735008 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:05.735035 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:05.735102 (+ 67us) server_negotiation.cc:407] Connection header received
0811 11:50:05.740040 (+ 4938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:05.740067 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:05.740530 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:05.740840 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:05.744509 (+ 3669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:05.745787 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:05.747545 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:05.748173 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:05.748309 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:05.753347 (+ 5038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:05.753392 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:05.753407 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:05.753441 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:05.755148 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:05.755798 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:05.755802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:05.755807 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:05.755943 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:05.756334 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:05.756338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:05.756339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:05.756936 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:05.757106 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:05.757972 (+ 866us) server_negotiation.cc:299] Negotiation successful
0811 11:50:05.758225 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":132,"threads_started":1}
I20260811 11:50:05.759796 4115 leader_election.cc:302] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51f9f6ca7ad24c0485f8e41b933675b4; no voters:
I20260811 11:50:05.760186 4115 leader_election.cc:288] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:50:05.760321 4124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:05.760510 4124 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4 [term 1 LEADER]: Becoming Leader. State: Replica: 51f9f6ca7ad24c0485f8e41b933675b4, State: Running, Role: LEADER
I20260811 11:50:05.760993 4124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51f9f6ca7ad24c0485f8e41b933675b4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36153 } }
I20260811 11:50:05.763574 4115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:50:05.764808 4127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51f9f6ca7ad24c0485f8e41b933675b4. Latest consensus state: current_term: 1 leader_uuid: "51f9f6ca7ad24c0485f8e41b933675b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51f9f6ca7ad24c0485f8e41b933675b4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36153 } } }
I20260811 11:50:05.764954 4127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:05.765394 4126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51f9f6ca7ad24c0485f8e41b933675b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51f9f6ca7ad24c0485f8e41b933675b4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36153 } } }
I20260811 11:50:05.765501 4126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:05.765894 4135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:50:05.771322 4135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:50:05.779371 4135 catalog_manager.cc:1347] Generated new cluster ID: 3ad19738789c4364940df5afaca404b9
I20260811 11:50:05.779454 4135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:50:05.795156 4135 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:50:05.796229 4135 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:50:05.813172 4135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51f9f6ca7ad24c0485f8e41b933675b4: Generated new TSK 0
I20260811 11:50:05.814138 4135 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
I20260811 11:50:05.852749 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36153
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:46701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:05.988512 4153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:05.988801 4153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:05.988852 4153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:05.993057 4153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:05.993129 4153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:05.993191 4153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:50:05.998762 4153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:46701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36153
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:06.000028 4153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:06.001089 4153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:06.011945 4165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:06.012148 4153 server_base.cc:1034] running on GCE node
W20260811 11:50:06.011958 4163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:06.011953 4161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:06.014513 4153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:06.015197 4153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:06.016398 4153 hybrid_clock.cc:648] HybridClock initialized: now 1786449006016377 us; error 36 us; skew 500 ppm
Aug 11 11:50:06 dist-test-slave-bndk krb5kdc[3965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449006, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:06.019719 4153 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:50:06.020665 4153 webserver.cc:460] Webserver started at http://127.3.221.65:43781/ using document root <none> and password file <none>
I20260811 11:50:06.021283 4153 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:06.021363 4153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:06.021579 4153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:06.023780 4153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "e2b66ae0c5ea4f2e953e16821efbbe1b"
format_stamp: "Formatted at 2026-08-11 11:50:06 on dist-test-slave-bndk"
server_key: "69c96f9b41c3e84da5731b213a15c866"
server_key_iv: "a50c05bf432a9a306c89754693a1fb20"
server_key_version: "encryptionkey@0"
I20260811 11:50:06.024410 4153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "e2b66ae0c5ea4f2e953e16821efbbe1b"
format_stamp: "Formatted at 2026-08-11 11:50:06 on dist-test-slave-bndk"
server_key: "69c96f9b41c3e84da5731b213a15c866"
server_key_iv: "a50c05bf432a9a306c89754693a1fb20"
server_key_version: "encryptionkey@0"
I20260811 11:50:06.028409 4153 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260811 11:50:06.031051 4182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:06.032373 4153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 11:50:06.032487 4153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "e2b66ae0c5ea4f2e953e16821efbbe1b"
format_stamp: "Formatted at 2026-08-11 11:50:06 on dist-test-slave-bndk"
server_key: "69c96f9b41c3e84da5731b213a15c866"
server_key_iv: "a50c05bf432a9a306c89754693a1fb20"
server_key_version: "encryptionkey@0"
I20260811 11:50:06.032589 4153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:06.059105 4153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:06.062247 4153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:06.062458 4153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:06.063203 4153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:06.064574 4153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:06.064640 4153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:06.064703 4153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:06.064733 4153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:06.075912 4153 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:46273
I20260811 11:50:06.075971 4374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:46273 every 8 connection(s)
I20260811 11:50:06.076871 4153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:50:06.081398 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 4153
I20260811 11:50:06.081492 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:50:06.081722 3957 external_mini_cluster.cc:1428] Setting key 43e345b16be9c2678f59310b103fe24c
Aug 11 11:50:06 dist-test-slave-bndk krb5kdc[3965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449006, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.221.66@KRBTEST.COM; defaulting to no policy
I20260811 11:50:06.092926 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:06.079486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:32969 (local address 127.3.221.126:36153)
0811 11:50:06.079674 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:06.079679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:06.080591 (+ 912us) server_negotiation.cc:407] Connection header received
0811 11:50:06.082295 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:06.082298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:06.082357 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:06.082446 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:06.083967 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:06.084722 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:06.086304 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:06.086547 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:06.086631 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:06.089423 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:06.089426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:06.089428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:06.089456 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:06.090870 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:06.091392 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:06.091395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:06.091396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:06.091477 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:06.091810 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:06.091812 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:06.091813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:06.091999 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:06.092084 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:06.092646 (+ 562us) server_negotiation.cc:299] Negotiation successful
0811 11:50:06.092791 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260811 11:50:06.093773 4377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:06.079903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36153 (local address 127.3.221.65:32969)
0811 11:50:06.080423 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:06.080470 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:06.082030 (+ 1560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:06.082659 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:06.082668 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:06.083229 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:06.083808 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:06.083825 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:06.084863 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:06.084865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:06.086147 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:06.086160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:06.086706 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:06.086710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:06.086957 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:06.087561 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:06.087581 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:06.089199 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:06.090982 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:06.090988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:06.091004 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:06.091284 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:06.091556 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:06.091559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:06.091561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:06.091702 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:06.092097 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:06.092104 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:06.092369 (+ 265us) client_negotiation.cc:769] Sending connection context
0811 11:50:06.092546 (+ 177us) client_negotiation.cc:240] Negotiation successful
0811 11:50:06.092833 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":126,"threads_started":1}
I20260811 11:50:06.095041 4375 heartbeater.cc:347] Connected to a master server at 127.3.221.126:36153
I20260811 11:50:06.095359 4375 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.66@KRBTEST.COM" created.
I20260811 11:50:06.095939 4375 heartbeater.cc:511] Master 127.3.221.126:36153 requested a full tablet report, sending...
I20260811 11:50:06.098402 4051 ts_manager.cc:194] Registered new tserver with Master: e2b66ae0c5ea4f2e953e16821efbbe1b (127.3.221.65:46273)
I20260811 11:50:06.100553 4051 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:32969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
I20260811 11:50:06.113032 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36153
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:46701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:06.232372 4381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:06.232640 4381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:06.232685 4381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:06.236800 4381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:06.236871 4381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:06.236931 4381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:50:06.242277 4381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:46701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36153
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:06.243306 4381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:06.244285 4381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:06.253811 4391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:06.253886 4390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:06.253897 4393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:06.254279 4381 server_base.cc:1034] running on GCE node
I20260811 11:50:06.254668 4381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:06.255285 4381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:06.256462 4381 hybrid_clock.cc:648] HybridClock initialized: now 1786449006256432 us; error 43 us; skew 500 ppm
Aug 11 11:50:06 dist-test-slave-bndk krb5kdc[3965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449006, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:06.259306 4381 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:50:06.260268 4381 webserver.cc:460] Webserver started at http://127.3.221.66:36617/ using document root <none> and password file <none>
I20260811 11:50:06.260847 4381 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:06.260911 4381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:06.261126 4381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:06.263341 4381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "a1365f72bb93429f92b12e27dd6257be"
format_stamp: "Formatted at 2026-08-11 11:50:06 on dist-test-slave-bndk"
server_key: "596f183ae6b3a3f69201f1421b6ebe23"
server_key_iv: "035f568a7ac5e7e33c75999c78b94f72"
server_key_version: "encryptionkey@0"
I20260811 11:50:06.263933 4381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "a1365f72bb93429f92b12e27dd6257be"
format_stamp: "Formatted at 2026-08-11 11:50:06 on dist-test-slave-bndk"
server_key: "596f183ae6b3a3f69201f1421b6ebe23"
server_key_iv: "035f568a7ac5e7e33c75999c78b94f72"
server_key_version: "encryptionkey@0"
I20260811 11:50:06.268030 4381 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260811 11:50:06.270787 4400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:06.272168 4381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 11:50:06.272295 4381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "a1365f72bb93429f92b12e27dd6257be"
format_stamp: "Formatted at 2026-08-11 11:50:06 on dist-test-slave-bndk"
server_key: "596f183ae6b3a3f69201f1421b6ebe23"
server_key_iv: "035f568a7ac5e7e33c75999c78b94f72"
server_key_version: "encryptionkey@0"
I20260811 11:50:06.272420 4381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:06.281824 4381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:06.284703 4381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:06.284888 4381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:06.285568 4381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:06.286499 4381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:06.286547 4381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:06.286592 4381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:06.286620 4381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:06.296665 4381 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:42039
I20260811 11:50:06.296687 4612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:42039 every 8 connection(s)
I20260811 11:50:06.297652 4381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:50:06.300695 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 4381
I20260811 11:50:06.300805 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:50:06.301075 3957 external_mini_cluster.cc:1428] Setting key 73453210cc9989dcb82bdb6831449409
Aug 11 11:50:06 dist-test-slave-bndk krb5kdc[3965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449006, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.221.67@KRBTEST.COM; defaulting to no policy
I20260811 11:50:06.314007 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:06.299620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:35997 (local address 127.3.221.126:36153)
0811 11:50:06.299749 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:06.299753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:06.300771 (+ 1018us) server_negotiation.cc:407] Connection header received
0811 11:50:06.302452 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:06.302455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:06.302505 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:06.302594 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:06.303988 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:06.304712 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:06.306484 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:06.306715 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:06.306809 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:06.309904 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:06.309908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:06.309910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:06.309940 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:06.311393 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:06.312038 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:06.312041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:06.312042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:06.312206 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:06.312667 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:06.312670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:06.312671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:06.312911 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:06.313012 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:06.313674 (+ 662us) server_negotiation.cc:299] Negotiation successful
0811 11:50:06.313830 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 11:50:06.314872 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:06.299994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36153 (local address 127.3.221.66:35997)
0811 11:50:06.300564 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:06.300624 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:06.302205 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:06.302840 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:06.302849 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:06.303286 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:06.303829 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:06.303845 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:06.304942 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:06.304946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:06.306353 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:06.306367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:06.306880 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:06.306885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:06.307086 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:06.307715 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:06.307737 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:06.309653 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:06.311530 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:06.311537 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:06.311556 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:06.311932 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:06.312310 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:06.312313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:06.312315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:06.312548 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:06.313026 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:06.313035 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:06.313375 (+ 340us) client_negotiation.cc:769] Sending connection context
0811 11:50:06.313593 (+ 218us) client_negotiation.cc:240] Negotiation successful
0811 11:50:06.313965 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":148,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.67@KRBTEST.COM" created.
I20260811 11:50:06.316373 4613 heartbeater.cc:347] Connected to a master server at 127.3.221.126:36153
I20260811 11:50:06.316674 4613 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:06.317368 4613 heartbeater.cc:511] Master 127.3.221.126:36153 requested a full tablet report, sending...
I20260811 11:50:06.318602 4051 ts_manager.cc:194] Registered new tserver with Master: a1365f72bb93429f92b12e27dd6257be (127.3.221.66:42039)
I20260811 11:50:06.319212 4051 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:35997
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
I20260811 11:50:06.334424 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36153
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:46701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:06.454558 4619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:06.455075 4619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:06.455188 4619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:06.460721 4619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:06.460794 4619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:06.460861 4619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:50:06.466528 4619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:46701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36153
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:06.467694 4619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:06.468770 4619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:06.478840 4626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:06.478966 4625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:06.478869 4630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:06.479122 4619 server_base.cc:1034] running on GCE node
I20260811 11:50:06.479633 4619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:06.480391 4619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:06.481554 4619 hybrid_clock.cc:648] HybridClock initialized: now 1786449006481530 us; error 30 us; skew 500 ppm
Aug 11 11:50:06 dist-test-slave-bndk krb5kdc[3965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449006, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:06.484431 4619 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:50:06.485409 4619 webserver.cc:460] Webserver started at http://127.3.221.67:45115/ using document root <none> and password file <none>
I20260811 11:50:06.486001 4619 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:06.486071 4619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:06.486245 4619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:06.488409 4619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "5c80387afdd44337b426ef47eb0d808a"
format_stamp: "Formatted at 2026-08-11 11:50:06 on dist-test-slave-bndk"
server_key: "5c9b445d60485c6873944fc56b9e2807"
server_key_iv: "da47d10b5949fb02b454a6f60a2fb87e"
server_key_version: "encryptionkey@0"
I20260811 11:50:06.488988 4619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "5c80387afdd44337b426ef47eb0d808a"
format_stamp: "Formatted at 2026-08-11 11:50:06 on dist-test-slave-bndk"
server_key: "5c9b445d60485c6873944fc56b9e2807"
server_key_iv: "da47d10b5949fb02b454a6f60a2fb87e"
server_key_version: "encryptionkey@0"
I20260811 11:50:06.493045 4619 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260811 11:50:06.495611 4640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:06.496937 4619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 11:50:06.497040 4619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "5c80387afdd44337b426ef47eb0d808a"
format_stamp: "Formatted at 2026-08-11 11:50:06 on dist-test-slave-bndk"
server_key: "5c9b445d60485c6873944fc56b9e2807"
server_key_iv: "da47d10b5949fb02b454a6f60a2fb87e"
server_key_version: "encryptionkey@0"
I20260811 11:50:06.497151 4619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:06.511658 4619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:06.514686 4619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:06.514848 4619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:06.515401 4619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:06.516227 4619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:06.516271 4619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:06.516309 4619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:06.516330 4619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:06.525792 4619 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:35843
I20260811 11:50:06.525856 4830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:35843 every 8 connection(s)
I20260811 11:50:06.526713 4619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:50:06.532341 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 4619
I20260811 11:50:06.532480 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:50:06.532763 3957 external_mini_cluster.cc:1428] Setting key 76b16e774a62764259be65ef41b4022d
Aug 11 11:50:06 dist-test-slave-bndk krb5kdc[3965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449006, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:06.540462 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:06.528683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:55595 (local address 127.3.221.126:36153)
0811 11:50:06.528856 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:06.528861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:06.529748 (+ 887us) server_negotiation.cc:407] Connection header received
0811 11:50:06.531098 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:06.531101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:06.531152 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:06.531235 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:06.532425 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:06.533272 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:06.534614 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:06.534838 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:06.534928 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:06.537554 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:06.537557 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:06.537559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:06.537585 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:06.538835 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:06.539253 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:06.539256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:06.539258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:06.539329 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:06.539577 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:06.539579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:06.539580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:06.539741 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:06.539810 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:06.540233 (+ 423us) server_negotiation.cc:299] Negotiation successful
0811 11:50:06.540327 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 11:50:06.541380 4833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:06.529068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36153 (local address 127.3.221.67:55595)
0811 11:50:06.529600 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:06.529644 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:06.530896 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:06.531410 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:06.531416 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:06.531779 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:06.532263 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:06.532276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:06.533426 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:06.533430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:06.534493 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:06.534504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:06.534983 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:06.534987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:06.535160 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:06.535802 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:06.535828 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:06.537385 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:06.538918 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:06.538923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:06.538935 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:06.539178 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:06.539388 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:06.539390 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:06.539392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:06.539514 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:06.539809 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:06.539815 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:06.540049 (+ 234us) client_negotiation.cc:769] Sending connection context
0811 11:50:06.540204 (+ 155us) client_negotiation.cc:240] Negotiation successful
0811 11:50:06.540463 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":167,"threads_started":1}
I20260811 11:50:06.542493 4831 heartbeater.cc:347] Connected to a master server at 127.3.221.126:36153
I20260811 11:50:06.542717 4831 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:06.543207 4831 heartbeater.cc:511] Master 127.3.221.126:36153 requested a full tablet report, sending...
I20260811 11:50:06.544185 4051 ts_manager.cc:194] Registered new tserver with Master: 5c80387afdd44337b426ef47eb0d808a (127.3.221.67:35843)
I20260811 11:50:06.544751 4051 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:55595
I20260811 11:50:06.547295 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 11:50:06 dist-test-slave-bndk krb5kdc[3965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449006, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 11:50:06 dist-test-slave-bndk krb5kdc[3965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449006, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.221.65@KRBTEST.COM
I20260811 11:50:06.575433 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:06.565754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.3.221.65:46273)
0811 11:50:06.566071 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:06.566089 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:06.566185 (+ 96us) server_negotiation.cc:407] Connection header received
0811 11:50:06.566351 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:06.566359 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:06.566559 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:06.566715 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:06.567443 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:06.568254 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:06.569088 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:06.569531 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:06.569635 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:06.572138 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:06.572148 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:06.572164 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:06.572194 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:06.573666 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:06.574086 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:06.574091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:06.574092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:06.574169 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:06.574468 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:06.574471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:06.574472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:06.574767 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:06.574964 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:06.575126 (+ 162us) server_negotiation.cc:299] Negotiation successful
0811 11:50:06.575273 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":116,"threads_started":1}
W20260811 11:50:06.576638 4240 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47424
Aug 11 11:50:06 dist-test-slave-bndk krb5kdc[3965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 11:50:06 dist-test-slave-bndk krb5kdc[3965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.65@KRBTEST.COM
I20260811 11:50:06.604151 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:06.595823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.3.221.65:46273)
0811 11:50:06.596002 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:06.596006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:06.596089 (+ 83us) server_negotiation.cc:407] Connection header received
0811 11:50:06.596211 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:06.596213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:06.596263 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:06.596341 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:06.597176 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:06.597768 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:06.598567 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:06.598808 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:06.598882 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:06.601285 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:06.601289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:06.601291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:06.601319 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:06.602636 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:06.603032 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:06.603034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:06.603035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:06.603110 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:06.603316 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:06.603318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:06.603319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:06.603486 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:06.603583 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:06.603860 (+ 277us) server_negotiation.cc:299] Negotiation successful
0811 11:50:06.603981 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 11:50:06.605361 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 4153
I20260811 11:50:06.623021 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 4381
I20260811 11:50:06.635655 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 4619
I20260811 11:50:06.644116 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 3989
2026-08-11T11:50:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2504 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:50:06 dist-test-slave-bndk krb5kdc[4849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:50:06 dist-test-slave-bndk krb5kdc[4849](info): set up 2 sockets
Aug 11 11:50:06 dist-test-slave-bndk krb5kdc[4849](info): commencing operation
krb5kdc: starting...
W20260811 11:50:07.814453 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:50:07 dist-test-slave-bndk krb5kdc[4849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:50:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:50:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:50:07.948922 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:33339
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:33339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:08.064988 4869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:08.065328 4869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:08.065385 4869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:08.069487 4869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:50:08.069561 4869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:08.069579 4869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:08.069594 4869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:50:08.069608 4869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:50:08.075258 4869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:33339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:33339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:08.076454 4869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:08.077565 4869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:08.086095 4884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:08.086089 4889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:08.086089 4881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:08.086390 4869 server_base.cc:1034] running on GCE node
I20260811 11:50:08.086958 4869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:08.088096 4869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:08.089282 4869 hybrid_clock.cc:648] HybridClock initialized: now 1786449008089266 us; error 33 us; skew 500 ppm
Aug 11 11:50:08 dist-test-slave-bndk krb5kdc[4849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449008, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:08.095312 4869 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:50:08.096276 4869 webserver.cc:460] Webserver started at http://127.3.221.126:37611/ using document root <none> and password file <none>
I20260811 11:50:08.096853 4869 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:08.096915 4869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:08.097128 4869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:08.099261 4869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "5b69c1dc2d9947ef87f255f1061fb1a6"
format_stamp: "Formatted at 2026-08-11 11:50:08 on dist-test-slave-bndk"
server_key: "5c5585e51d648fa702d9dc6e704d9dbc"
server_key_iv: "02d93fd9ab2ccda2e573d22b62f63404"
server_key_version: "encryptionkey@0"
I20260811 11:50:08.099840 4869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "5b69c1dc2d9947ef87f255f1061fb1a6"
format_stamp: "Formatted at 2026-08-11 11:50:08 on dist-test-slave-bndk"
server_key: "5c5585e51d648fa702d9dc6e704d9dbc"
server_key_iv: "02d93fd9ab2ccda2e573d22b62f63404"
server_key_version: "encryptionkey@0"
I20260811 11:50:08.103683 4869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 11:50:08.106241 4904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:08.107532 4869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 11:50:08.107656 4869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "5b69c1dc2d9947ef87f255f1061fb1a6"
format_stamp: "Formatted at 2026-08-11 11:50:08 on dist-test-slave-bndk"
server_key: "5c5585e51d648fa702d9dc6e704d9dbc"
server_key_iv: "02d93fd9ab2ccda2e573d22b62f63404"
server_key_version: "encryptionkey@0"
I20260811 11:50:08.107764 4869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:08.123149 4869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:08.126364 4869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:08.126564 4869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:08.134286 4869 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:33339
I20260811 11:50:08.134361 4994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:33339 every 8 connection(s)
I20260811 11:50:08.135336 4869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:50:08.136170 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 4869
I20260811 11:50:08.136270 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:50:08.136541 3957 external_mini_cluster.cc:1428] Setting key 767fafcf374ea58d28f3f6445a67b796
I20260811 11:50:08.138717 4995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:08.144675 4995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6: Bootstrap starting.
Aug 11 11:50:08 dist-test-slave-bndk krb5kdc[4849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:08.147316 4995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:08.148075 4995 log.cc:824] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:08.150377 4995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6: No bootstrap required, opened a new log
I20260811 11:50:08.150581 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:08.137988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38424 (local address 127.3.221.126:33339)
0811 11:50:08.138443 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:08.138459 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:08.138495 (+ 36us) server_negotiation.cc:407] Connection header received
0811 11:50:08.139187 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:08.139206 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:08.139554 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:08.139873 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:08.140682 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.141748 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.142578 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.143050 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.143194 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:08.145899 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:08.145905 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:08.145918 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:08.145961 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:08.147570 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:08.148000 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:08.148005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:08.148011 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:08.148126 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:08.148477 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:08.148479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:08.148480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:08.149030 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:08.149175 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:08.149502 (+ 327us) server_negotiation.cc:299] Negotiation successful
0811 11:50:08.149796 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":156,"threads_started":1}
I20260811 11:50:08.154704 4995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b69c1dc2d9947ef87f255f1061fb1a6" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33339 } }
I20260811 11:50:08.154959 4995 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:08.155025 4995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b69c1dc2d9947ef87f255f1061fb1a6, State: Initialized, Role: FOLLOWER
I20260811 11:50:08.155489 4995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b69c1dc2d9947ef87f255f1061fb1a6" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33339 } }
I20260811 11:50:08.155629 4995 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:50:08.155694 4995 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:50:08.155793 4995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:08.156777 4995 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b69c1dc2d9947ef87f255f1061fb1a6" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33339 } }
I20260811 11:50:08.157127 4995 leader_election.cc:302] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b69c1dc2d9947ef87f255f1061fb1a6; no voters:
I20260811 11:50:08.157526 4995 leader_election.cc:288] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:50:08.157657 5005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:08.157888 5005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5b69c1dc2d9947ef87f255f1061fb1a6, State: Running, Role: LEADER
I20260811 11:50:08.158229 5005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b69c1dc2d9947ef87f255f1061fb1a6" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33339 } }
I20260811 11:50:08.158870 4995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:50:08.160037 5007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b69c1dc2d9947ef87f255f1061fb1a6. Latest consensus state: current_term: 1 leader_uuid: "5b69c1dc2d9947ef87f255f1061fb1a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b69c1dc2d9947ef87f255f1061fb1a6" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33339 } } }
I20260811 11:50:08.160194 5007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:08.160475 5006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b69c1dc2d9947ef87f255f1061fb1a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b69c1dc2d9947ef87f255f1061fb1a6" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33339 } } }
I20260811 11:50:08.160593 5006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:08.160694 5017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:50:08.163985 5017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:50:08.170722 5017 catalog_manager.cc:1347] Generated new cluster ID: 400d82114346468d982798492051d59d
I20260811 11:50:08.170800 5017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:50:08.196626 5017 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:50:08.197659 5017 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:50:08.205978 5017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b69c1dc2d9947ef87f255f1061fb1a6: Generated new TSK 0
I20260811 11:50:08.206750 5017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
I20260811 11:50:08.247926 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:33339
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:08.366355 5037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:08.366643 5037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:08.366690 5037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:08.370810 5037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:08.370880 5037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:08.370946 5037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:50:08.376714 5037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:33339
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:08.377961 5037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:08.379065 5037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:08.389039 5045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:08.389039 5049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:08.389086 5043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:08.389200 5037 server_base.cc:1034] running on GCE node
I20260811 11:50:08.389880 5037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:08.390578 5037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:08.391760 5037 hybrid_clock.cc:648] HybridClock initialized: now 1786449008391738 us; error 33 us; skew 500 ppm
Aug 11 11:50:08 dist-test-slave-bndk krb5kdc[4849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449008, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:08.394655 5037 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:50:08.395607 5037 webserver.cc:460] Webserver started at http://127.3.221.65:42853/ using document root <none> and password file <none>
I20260811 11:50:08.396191 5037 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:08.396265 5037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:08.396497 5037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:08.398726 5037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "09bbac8eb78a487c9e69c3820ae3cdf7"
format_stamp: "Formatted at 2026-08-11 11:50:08 on dist-test-slave-bndk"
server_key: "f054d2db64a8394585e234d85d978f91"
server_key_iv: "dc9c19a39231104bccccaedfd7291aa2"
server_key_version: "encryptionkey@0"
I20260811 11:50:08.399343 5037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "09bbac8eb78a487c9e69c3820ae3cdf7"
format_stamp: "Formatted at 2026-08-11 11:50:08 on dist-test-slave-bndk"
server_key: "f054d2db64a8394585e234d85d978f91"
server_key_iv: "dc9c19a39231104bccccaedfd7291aa2"
server_key_version: "encryptionkey@0"
I20260811 11:50:08.403677 5037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 11:50:08.406441 5059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:08.407975 5037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:50:08.408110 5037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "09bbac8eb78a487c9e69c3820ae3cdf7"
format_stamp: "Formatted at 2026-08-11 11:50:08 on dist-test-slave-bndk"
server_key: "f054d2db64a8394585e234d85d978f91"
server_key_iv: "dc9c19a39231104bccccaedfd7291aa2"
server_key_version: "encryptionkey@0"
I20260811 11:50:08.408229 5037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:08.444311 5037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:08.447505 5037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:08.447707 5037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:08.448359 5037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:08.449254 5037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:08.449303 5037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:08.449361 5037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:08.449390 5037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:08.458148 5037 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:41861
I20260811 11:50:08.458177 5240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:41861 every 8 connection(s)
I20260811 11:50:08.459158 5037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:50:08.466071 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 5037
I20260811 11:50:08.466197 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:50:08.466445 3957 external_mini_cluster.cc:1428] Setting key da7ef8f14e82136fafc81ef277bda5bb
Aug 11 11:50:08 dist-test-slave-bndk krb5kdc[4849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449008, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:08.474390 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:08.461121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:59431 (local address 127.3.221.126:33339)
0811 11:50:08.461295 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:08.461299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:08.462206 (+ 907us) server_negotiation.cc:407] Connection header received
0811 11:50:08.463656 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:08.463659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:08.463710 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:08.463806 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:08.465033 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.465655 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.467015 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.467315 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.467403 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:08.470411 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:08.470416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:08.470419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:08.470452 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:08.471954 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:08.472578 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:08.472581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:08.472582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:08.472660 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:08.473034 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:08.473036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:08.473038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:08.473248 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:08.473334 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:08.474087 (+ 753us) server_negotiation.cc:299] Negotiation successful
0811 11:50:08.474246 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 11:50:08.475598 5245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:08.461569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:33339 (local address 127.3.221.65:59431)
0811 11:50:08.462039 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:08.462084 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:08.463401 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:08.463969 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:08.463976 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:08.464340 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:08.464859 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:08.464873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.465805 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.465808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:08.466874 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:08.466887 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.467506 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.467511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:08.467742 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:08.468498 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:08.468516 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:08.470226 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:08.472085 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:08.472092 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:08.472113 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:08.472478 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:08.472746 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:08.472750 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:08.472752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:08.472951 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:08.473365 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:08.473374 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:08.473749 (+ 375us) client_negotiation.cc:769] Sending connection context
0811 11:50:08.474006 (+ 257us) client_negotiation.cc:240] Negotiation successful
0811 11:50:08.474428 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":121,"threads_started":1}
I20260811 11:50:08.477102 5241 heartbeater.cc:347] Connected to a master server at 127.3.221.126:33339
I20260811 11:50:08.477401 5241 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.221.66@KRBTEST.COM; defaulting to no policy
I20260811 11:50:08.477931 5241 heartbeater.cc:511] Master 127.3.221.126:33339 requested a full tablet report, sending...
I20260811 11:50:08.479545 4929 ts_manager.cc:194] Registered new tserver with Master: 09bbac8eb78a487c9e69c3820ae3cdf7 (127.3.221.65:41861)
I20260811 11:50:08.480789 4929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:59431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
I20260811 11:50:08.498421 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:33339
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:08.615464 5251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:08.615757 5251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:08.615823 5251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:08.619875 5251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:08.619952 5251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:08.620038 5251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:50:08.625454 5251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:33339
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:08.626595 5251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:08.627669 5251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:08.637728 5260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:08.637902 5251 server_base.cc:1034] running on GCE node
W20260811 11:50:08.637764 5262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:08.638015 5266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:08.638448 5251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:08.639082 5251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:08.640239 5251 hybrid_clock.cc:648] HybridClock initialized: now 1786449008640196 us; error 56 us; skew 500 ppm
Aug 11 11:50:08 dist-test-slave-bndk krb5kdc[4849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449008, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:08.643060 5251 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:50:08.643958 5251 webserver.cc:460] Webserver started at http://127.3.221.66:40467/ using document root <none> and password file <none>
I20260811 11:50:08.644519 5251 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:08.644582 5251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:08.644788 5251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:08.646955 5251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "2dfa16d7bad9491eb7545fd03602ba25"
format_stamp: "Formatted at 2026-08-11 11:50:08 on dist-test-slave-bndk"
server_key: "3eccb68d8ac2b420b861a46a4d3e13b9"
server_key_iv: "2e77fe7eea3ff49a93d14c04ea90c205"
server_key_version: "encryptionkey@0"
I20260811 11:50:08.647519 5251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "2dfa16d7bad9491eb7545fd03602ba25"
format_stamp: "Formatted at 2026-08-11 11:50:08 on dist-test-slave-bndk"
server_key: "3eccb68d8ac2b420b861a46a4d3e13b9"
server_key_iv: "2e77fe7eea3ff49a93d14c04ea90c205"
server_key_version: "encryptionkey@0"
I20260811 11:50:08.651470 5251 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 11:50:08.653900 5279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:08.655097 5251 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 11:50:08.655206 5251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "2dfa16d7bad9491eb7545fd03602ba25"
format_stamp: "Formatted at 2026-08-11 11:50:08 on dist-test-slave-bndk"
server_key: "3eccb68d8ac2b420b861a46a4d3e13b9"
server_key_iv: "2e77fe7eea3ff49a93d14c04ea90c205"
server_key_version: "encryptionkey@0"
I20260811 11:50:08.655304 5251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:08.673491 5251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:08.676471 5251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:08.676666 5251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:08.677327 5251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:08.678225 5251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:08.678274 5251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:08.678330 5251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:08.678368 5251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:08.687520 5251 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:35345
I20260811 11:50:08.687587 5458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:35345 every 8 connection(s)
I20260811 11:50:08.688467 5251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:50:08.696813 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 5251
I20260811 11:50:08.696946 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:50:08.697202 3957 external_mini_cluster.cc:1428] Setting key 14e69ca7a0e89e0a924b8e4067143993
Aug 11 11:50:08 dist-test-slave-bndk krb5kdc[4849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449008, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:08.702358 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:08.690372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:44829 (local address 127.3.221.126:33339)
0811 11:50:08.690548 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:08.690553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:08.691520 (+ 967us) server_negotiation.cc:407] Connection header received
0811 11:50:08.692821 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:08.692824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:08.692887 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:08.692963 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:08.694267 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.694794 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.695880 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.696089 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.696159 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:08.698788 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:08.698791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:08.698793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:08.698819 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:08.700157 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:08.700770 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:08.700773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:08.700774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:08.700862 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:08.701192 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:08.701196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:08.701197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:08.701425 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:08.701516 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:08.702045 (+ 529us) server_negotiation.cc:299] Negotiation successful
0811 11:50:08.702178 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 11:50:08.703219 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:08.690802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:33339 (local address 127.3.221.66:44829)
0811 11:50:08.691353 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:08.691400 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:08.692620 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:08.693172 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:08.693179 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:08.693632 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:08.694136 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:08.694151 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.694902 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.694905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:08.695774 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:08.695784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.696205 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.696208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:08.696372 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:08.697034 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:08.697050 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:08.698622 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:08.700284 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:08.700291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:08.700308 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:08.700665 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:08.700954 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:08.700957 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:08.700959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:08.701097 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:08.701544 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:08.701552 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:08.701805 (+ 253us) client_negotiation.cc:769] Sending connection context
0811 11:50:08.701981 (+ 176us) client_negotiation.cc:240] Negotiation successful
0811 11:50:08.702317 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":124,"threads_started":1}
I20260811 11:50:08.704559 5459 heartbeater.cc:347] Connected to a master server at 127.3.221.126:33339
I20260811 11:50:08.704793 5459 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:08.705333 5459 heartbeater.cc:511] Master 127.3.221.126:33339 requested a full tablet report, sending...
I20260811 11:50:08.706486 4929 ts_manager.cc:194] Registered new tserver with Master: 2dfa16d7bad9491eb7545fd03602ba25 (127.3.221.66:35345)
I20260811 11:50:08.707261 4929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:44829
WARNING: no policy specified for kudu/127.3.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
I20260811 11:50:08.728339 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:33339
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:08.845546 5466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:08.845818 5466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:08.845862 5466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:08.849852 5466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:08.849920 5466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:08.849978 5466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:50:08.855376 5466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:33339
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:08.856442 5466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:08.857481 5466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:08.867800 5482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:08.867818 5466 server_base.cc:1034] running on GCE node
W20260811 11:50:08.867857 5477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:08.867832 5476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:08.868453 5466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:08.869133 5466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:08.870389 5466 hybrid_clock.cc:648] HybridClock initialized: now 1786449008870367 us; error 34 us; skew 500 ppm
Aug 11 11:50:08 dist-test-slave-bndk krb5kdc[4849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449008, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:08.873137 5466 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:50:08.874084 5466 webserver.cc:460] Webserver started at http://127.3.221.67:35225/ using document root <none> and password file <none>
I20260811 11:50:08.874665 5466 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:08.874737 5466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:08.874958 5466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:08.877100 5466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "8a176fbfcbdd440aa4a9fb9fa319262d"
format_stamp: "Formatted at 2026-08-11 11:50:08 on dist-test-slave-bndk"
server_key: "8520f835a8fb6d788f0d2639530c7f02"
server_key_iv: "8f40a94a0e2f87ac19996fff5a21bc29"
server_key_version: "encryptionkey@0"
I20260811 11:50:08.877760 5466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "8a176fbfcbdd440aa4a9fb9fa319262d"
format_stamp: "Formatted at 2026-08-11 11:50:08 on dist-test-slave-bndk"
server_key: "8520f835a8fb6d788f0d2639530c7f02"
server_key_iv: "8f40a94a0e2f87ac19996fff5a21bc29"
server_key_version: "encryptionkey@0"
I20260811 11:50:08.881821 5466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 11:50:08.884536 5495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:08.885957 5466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:50:08.886086 5466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "8a176fbfcbdd440aa4a9fb9fa319262d"
format_stamp: "Formatted at 2026-08-11 11:50:08 on dist-test-slave-bndk"
server_key: "8520f835a8fb6d788f0d2639530c7f02"
server_key_iv: "8f40a94a0e2f87ac19996fff5a21bc29"
server_key_version: "encryptionkey@0"
I20260811 11:50:08.886205 5466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:08.905123 5466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:08.908358 5466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:08.908540 5466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:08.909116 5466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:08.909981 5466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:08.910029 5466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:08.910068 5466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:08.910087 5466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:08.918887 5466 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:38747
I20260811 11:50:08.918939 5674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:38747 every 8 connection(s)
I20260811 11:50:08.919817 5466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:50:08.925904 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 5466
I20260811 11:50:08.926040 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:50:08.926345 3957 external_mini_cluster.cc:1428] Setting key af0ad21f82d14752a5270c1379265528
Aug 11 11:50:08 dist-test-slave-bndk krb5kdc[4849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449008, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:08.933770 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:08.921683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:55127 (local address 127.3.221.126:33339)
0811 11:50:08.921833 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:08.921838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:08.922719 (+ 881us) server_negotiation.cc:407] Connection header received
0811 11:50:08.924002 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:08.924005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:08.924054 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:08.924126 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:08.925266 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.925840 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.927333 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.927585 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.927687 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:08.930277 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:08.930280 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:08.930282 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:08.930310 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:08.931655 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:08.932241 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:08.932244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:08.932245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:08.932353 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:08.932707 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:08.932710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:08.932711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:08.932921 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:08.933010 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:08.933521 (+ 511us) server_negotiation.cc:299] Negotiation successful
0811 11:50:08.933637 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 11:50:08.934595 5680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:08.922080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:33339 (local address 127.3.221.67:55127)
0811 11:50:08.922540 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:08.922585 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:08.923794 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:08.924264 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:08.924270 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:08.924632 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:08.925117 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:08.925130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.925970 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.925975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:08.927215 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:08.927227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.927785 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.927790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:08.927973 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:08.928599 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:08.928616 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:08.930113 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:08.931820 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:08.931826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:08.931841 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:08.932128 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:08.932436 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:08.932439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:08.932440 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:08.932597 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:08.933010 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:08.933015 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:08.933284 (+ 269us) client_negotiation.cc:769] Sending connection context
0811 11:50:08.933446 (+ 162us) client_negotiation.cc:240] Negotiation successful
0811 11:50:08.933720 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":110,"threads_started":1}
I20260811 11:50:08.935751 5677 heartbeater.cc:347] Connected to a master server at 127.3.221.126:33339
I20260811 11:50:08.936125 5677 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:08.936671 5677 heartbeater.cc:511] Master 127.3.221.126:33339 requested a full tablet report, sending...
I20260811 11:50:08.937879 4929 ts_manager.cc:194] Registered new tserver with Master: 8a176fbfcbdd440aa4a9fb9fa319262d (127.3.221.67:38747)
I20260811 11:50:08.938428 4929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:55127
I20260811 11:50:08.940405 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:50:08.952081 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:08.945006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38430 (local address 127.3.221.126:33339)
0811 11:50:08.945172 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:08.945177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:08.945310 (+ 133us) server_negotiation.cc:407] Connection header received
0811 11:50:08.945464 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:08.945468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:08.945515 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:08.945582 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:08.946310 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.946828 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.947676 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.947900 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.947990 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:08.949331 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:08.949335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:08.949338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:08.949364 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:08.950414 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:08.950812 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:08.950816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:08.950818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:08.950906 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:08.951166 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:08.951169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:08.951171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:08.951424 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:08.951548 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:08.951795 (+ 247us) server_negotiation.cc:299] Negotiation successful
0811 11:50:08.951922 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260811 11:50:08.958102 4929 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:50:08.961995 4929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:50:08.977774 5694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:08.972160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:41861 (local address 127.0.0.1:58048)
0811 11:50:08.972601 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:08.972655 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:08.972793 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:08.973714 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:08.973722 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:08.973747 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:08.973912 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:08.973919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.975150 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.975153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:08.976445 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:08.976456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.977337 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.977340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:08.977426 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:08.977461 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 11:50:08.977541 (+ 80us) client_negotiation.cc:240] Negotiation successful
0811 11:50:08.977632 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":6,"thread_start_us":102,"threads_started":1}
I20260811 11:50:08.977895 5696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:08.972286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58048 (local address 127.3.221.65:41861)
0811 11:50:08.973097 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:08.973115 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:08.973140 (+ 25us) server_negotiation.cc:407] Connection header received
0811 11:50:08.973253 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:08.973259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:08.973519 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:08.974063 (+ 544us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:08.974218 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.975023 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.976563 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.977238 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.977350 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:08.977582 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:08.977644 (+ 62us) server_negotiation.cc:299] Negotiation successful
0811 11:50:08.977744 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":86,"threads_started":1}
I20260811 11:50:08.978557 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:08.972938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:35345 (local address 127.0.0.1:35200)
0811 11:50:08.973675 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:08.973687 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:08.973764 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:08.974172 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:08.974175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:08.974192 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:08.974272 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:08.974277 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.975559 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.975562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:08.976951 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:08.976963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.978184 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.978188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:08.978266 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:08.978285 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 11:50:08.978352 (+ 67us) client_negotiation.cc:240] Negotiation successful
0811 11:50:08.978423 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":56,"thread_start_us":84,"threads_started":1}
I20260811 11:50:08.978839 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:08.973361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35200 (local address 127.3.221.66:35345)
0811 11:50:08.973775 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:08.973786 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:08.973807 (+ 21us) server_negotiation.cc:407] Connection header received
0811 11:50:08.973880 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:08.973885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:08.974049 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:08.974189 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:08.974359 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.975435 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.977108 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.977954 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.978058 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:08.978199 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:08.978413 (+ 214us) server_negotiation.cc:299] Negotiation successful
0811 11:50:08.978509 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":135,"threads_started":1}
I20260811 11:50:08.979616 5702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:08.972482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:38747 (local address 127.0.0.1:41476)
0811 11:50:08.973399 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:08.973413 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:08.973494 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:08.973915 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:08.973918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:08.973934 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:08.974015 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:08.974021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.975509 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.975512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:08.976951 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:08.976963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.979219 (+ 2256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.979222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:08.979315 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:08.979338 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 11:50:08.979426 (+ 88us) client_negotiation.cc:240] Negotiation successful
0811 11:50:08.979493 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"mutex_wait_us":151,"thread_start_us":104,"threads_started":1}
I20260811 11:50:08.980183 5704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:08.973007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41476 (local address 127.3.221.67:38747)
0811 11:50:08.973466 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:08.973479 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:08.973505 (+ 26us) server_negotiation.cc:407] Connection header received
0811 11:50:08.973587 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:08.973592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:08.973775 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:08.973943 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:08.974106 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.975355 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.978267 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:08.979088 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:08.979208 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:08.979351 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:08.979935 (+ 584us) server_negotiation.cc:299] Negotiation successful
0811 11:50:08.980037 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":99,"threads_started":1}
I20260811 11:50:08.981630 5143 tablet_service.cc:1467] Processing CreateTablet for tablet 3e370aab5a3c4c3f9e6617ca67485a73 (DEFAULT_TABLE table=test-table [id=71f2a957dfaa43adb290e21e262dd641]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:08.981659 5361 tablet_service.cc:1467] Processing CreateTablet for tablet 3e370aab5a3c4c3f9e6617ca67485a73 (DEFAULT_TABLE table=test-table [id=71f2a957dfaa43adb290e21e262dd641]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:08.982803 5143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e370aab5a3c4c3f9e6617ca67485a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:08.982842 5361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e370aab5a3c4c3f9e6617ca67485a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:08.983000 5574 tablet_service.cc:1467] Processing CreateTablet for tablet 3e370aab5a3c4c3f9e6617ca67485a73 (DEFAULT_TABLE table=test-table [id=71f2a957dfaa43adb290e21e262dd641]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:08.983958 5574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e370aab5a3c4c3f9e6617ca67485a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:08.989781 5706 tablet_bootstrap.cc:492] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7: Bootstrap starting.
I20260811 11:50:08.990803 5708 tablet_bootstrap.cc:492] T 3e370aab5a3c4c3f9e6617ca67485a73 P 2dfa16d7bad9491eb7545fd03602ba25: Bootstrap starting.
I20260811 11:50:08.992367 5706 tablet_bootstrap.cc:654] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:08.993142 5706 log.cc:824] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:08.993418 5708 tablet_bootstrap.cc:654] T 3e370aab5a3c4c3f9e6617ca67485a73 P 2dfa16d7bad9491eb7545fd03602ba25: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:08.993667 5709 tablet_bootstrap.cc:492] T 3e370aab5a3c4c3f9e6617ca67485a73 P 8a176fbfcbdd440aa4a9fb9fa319262d: Bootstrap starting.
I20260811 11:50:08.994310 5708 log.cc:824] T 3e370aab5a3c4c3f9e6617ca67485a73 P 2dfa16d7bad9491eb7545fd03602ba25: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:08.995402 5706 tablet_bootstrap.cc:492] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7: No bootstrap required, opened a new log
I20260811 11:50:08.995615 5706 ts_tablet_manager.cc:1397] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 11:50:08.996582 5709 tablet_bootstrap.cc:654] T 3e370aab5a3c4c3f9e6617ca67485a73 P 8a176fbfcbdd440aa4a9fb9fa319262d: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:08.996805 5708 tablet_bootstrap.cc:492] T 3e370aab5a3c4c3f9e6617ca67485a73 P 2dfa16d7bad9491eb7545fd03602ba25: No bootstrap required, opened a new log
I20260811 11:50:08.997011 5708 ts_tablet_manager.cc:1397] T 3e370aab5a3c4c3f9e6617ca67485a73 P 2dfa16d7bad9491eb7545fd03602ba25: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260811 11:50:08.997524 5709 log.cc:824] T 3e370aab5a3c4c3f9e6617ca67485a73 P 8a176fbfcbdd440aa4a9fb9fa319262d: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:08.999709 5709 tablet_bootstrap.cc:492] T 3e370aab5a3c4c3f9e6617ca67485a73 P 8a176fbfcbdd440aa4a9fb9fa319262d: No bootstrap required, opened a new log
I20260811 11:50:08.999639 5706 raft_consensus.cc:348] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bbac8eb78a487c9e69c3820ae3cdf7" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41861 } } peers { permanent_uuid: "2dfa16d7bad9491eb7545fd03602ba25" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 35345 } } peers { permanent_uuid: "8a176fbfcbdd440aa4a9fb9fa319262d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 38747 } }
I20260811 11:50:08.999902 5706 raft_consensus.cc:374] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:08.999923 5709 ts_tablet_manager.cc:1397] T 3e370aab5a3c4c3f9e6617ca67485a73 P 8a176fbfcbdd440aa4a9fb9fa319262d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 11:50:08.999955 5706 raft_consensus.cc:729] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09bbac8eb78a487c9e69c3820ae3cdf7, State: Initialized, Role: FOLLOWER
I20260811 11:50:09.000408 5706 consensus_queue.cc:260] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bbac8eb78a487c9e69c3820ae3cdf7" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41861 } } peers { permanent_uuid: "2dfa16d7bad9491eb7545fd03602ba25" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 35345 } } peers { permanent_uuid: "8a176fbfcbdd440aa4a9fb9fa319262d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 38747 } }
I20260811 11:50:09.000818 5708 raft_consensus.cc:348] T 3e370aab5a3c4c3f9e6617ca67485a73 P 2dfa16d7bad9491eb7545fd03602ba25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bbac8eb78a487c9e69c3820ae3cdf7" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41861 } } peers { permanent_uuid: "2dfa16d7bad9491eb7545fd03602ba25" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 35345 } } peers { permanent_uuid: "8a176fbfcbdd440aa4a9fb9fa319262d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 38747 } }
I20260811 11:50:09.001102 5708 raft_consensus.cc:374] T 3e370aab5a3c4c3f9e6617ca67485a73 P 2dfa16d7bad9491eb7545fd03602ba25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:09.001194 5241 heartbeater.cc:503] Master 127.3.221.126:33339 was elected leader, sending a full tablet report...
I20260811 11:50:09.001509 5706 ts_tablet_manager.cc:1428] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 11:50:09.001149 5708 raft_consensus.cc:729] T 3e370aab5a3c4c3f9e6617ca67485a73 P 2dfa16d7bad9491eb7545fd03602ba25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dfa16d7bad9491eb7545fd03602ba25, State: Initialized, Role: FOLLOWER
I20260811 11:50:09.002146 5708 consensus_queue.cc:260] T 3e370aab5a3c4c3f9e6617ca67485a73 P 2dfa16d7bad9491eb7545fd03602ba25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bbac8eb78a487c9e69c3820ae3cdf7" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41861 } } peers { permanent_uuid: "2dfa16d7bad9491eb7545fd03602ba25" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 35345 } } peers { permanent_uuid: "8a176fbfcbdd440aa4a9fb9fa319262d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 38747 } }
I20260811 11:50:09.003021 5459 heartbeater.cc:503] Master 127.3.221.126:33339 was elected leader, sending a full tablet report...
I20260811 11:50:09.003335 5708 ts_tablet_manager.cc:1428] T 3e370aab5a3c4c3f9e6617ca67485a73 P 2dfa16d7bad9491eb7545fd03602ba25: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260811 11:50:09.003827 5709 raft_consensus.cc:348] T 3e370aab5a3c4c3f9e6617ca67485a73 P 8a176fbfcbdd440aa4a9fb9fa319262d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bbac8eb78a487c9e69c3820ae3cdf7" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41861 } } peers { permanent_uuid: "2dfa16d7bad9491eb7545fd03602ba25" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 35345 } } peers { permanent_uuid: "8a176fbfcbdd440aa4a9fb9fa319262d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 38747 } }
I20260811 11:50:09.004056 5709 raft_consensus.cc:374] T 3e370aab5a3c4c3f9e6617ca67485a73 P 8a176fbfcbdd440aa4a9fb9fa319262d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:09.004118 5709 raft_consensus.cc:729] T 3e370aab5a3c4c3f9e6617ca67485a73 P 8a176fbfcbdd440aa4a9fb9fa319262d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a176fbfcbdd440aa4a9fb9fa319262d, State: Initialized, Role: FOLLOWER
I20260811 11:50:09.004565 5709 consensus_queue.cc:260] T 3e370aab5a3c4c3f9e6617ca67485a73 P 8a176fbfcbdd440aa4a9fb9fa319262d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bbac8eb78a487c9e69c3820ae3cdf7" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41861 } } peers { permanent_uuid: "2dfa16d7bad9491eb7545fd03602ba25" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 35345 } } peers { permanent_uuid: "8a176fbfcbdd440aa4a9fb9fa319262d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 38747 } }
I20260811 11:50:09.005336 5677 heartbeater.cc:503] Master 127.3.221.126:33339 was elected leader, sending a full tablet report...
I20260811 11:50:09.005609 5709 ts_tablet_manager.cc:1428] T 3e370aab5a3c4c3f9e6617ca67485a73 P 8a176fbfcbdd440aa4a9fb9fa319262d: Time spent starting tablet: real 0.006s user 0.001s sys 0.006s
W20260811 11:50:09.170877 5678 tablet.cc:2369] T 3e370aab5a3c4c3f9e6617ca67485a73 P 8a176fbfcbdd440aa4a9fb9fa319262d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:50:09.189728 5460 tablet.cc:2369] T 3e370aab5a3c4c3f9e6617ca67485a73 P 2dfa16d7bad9491eb7545fd03602ba25: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:50:09.210628 5242 tablet.cc:2369] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:50:09.440876 5716 raft_consensus.cc:482] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:50:09.441103 5716 raft_consensus.cc:504] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bbac8eb78a487c9e69c3820ae3cdf7" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41861 } } peers { permanent_uuid: "2dfa16d7bad9491eb7545fd03602ba25" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 35345 } } peers { permanent_uuid: "8a176fbfcbdd440aa4a9fb9fa319262d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 38747 } }
I20260811 11:50:09.442394 5716 leader_election.cc:288] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2dfa16d7bad9491eb7545fd03602ba25 (127.3.221.66:35345), 8a176fbfcbdd440aa4a9fb9fa319262d (127.3.221.67:38747)
I20260811 11:50:09.446393 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:09.442691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:48333 (local address 127.3.221.66:35345)
0811 11:50:09.442850 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:09.442854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:09.443126 (+ 272us) server_negotiation.cc:407] Connection header received
0811 11:50:09.443319 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:09.443322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:09.443368 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:09.443440 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:09.443727 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:09.444228 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:09.445464 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:09.445848 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:09.445953 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:09.446008 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:09.446176 (+ 168us) server_negotiation.cc:299] Negotiation successful
0811 11:50:09.446247 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 11:50:09.446409 5704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:09.442742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:34893 (local address 127.3.221.67:38747)
0811 11:50:09.442881 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:09.442885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:09.443145 (+ 260us) server_negotiation.cc:407] Connection header received
0811 11:50:09.443306 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:09.443308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:09.443354 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:09.443413 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:09.443757 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:09.444246 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:09.445480 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:09.445862 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:09.445971 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:09.446022 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:09.446211 (+ 189us) server_negotiation.cc:299] Negotiation successful
0811 11:50:09.446293 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 11:50:09.446434 5722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:09.442641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:35345 (local address 127.3.221.65:48333)
0811 11:50:09.443008 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:09.443021 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:09.443183 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:09.443465 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:09.443468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:09.443484 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:09.443617 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:09.443623 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:09.444337 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:09.444340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:09.445340 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:09.445351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:09.445964 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:09.445968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:09.446060 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:09.446084 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 11:50:09.446145 (+ 61us) client_negotiation.cc:240] Negotiation successful
0811 11:50:09.446214 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":65,"spinlock_wait_cycles":37888,"thread_start_us":174,"threads_started":1}
I20260811 11:50:09.446437 5721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:09.442660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:38747 (local address 127.3.221.65:34893)
0811 11:50:09.443025 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:09.443037 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:09.443181 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:09.443448 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:09.443451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:09.443467 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:09.443638 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:09.443644 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:09.444357 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:09.444361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:09.445347 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:09.445358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:09.445977 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:09.445981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:09.446088 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:09.446108 (+ 20us) client_negotiation.cc:769] Sending connection context
0811 11:50:09.446180 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 11:50:09.446247 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":44,"spinlock_wait_cycles":26624,"thread_start_us":144,"threads_started":1}
I20260811 11:50:09.447197 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e370aab5a3c4c3f9e6617ca67485a73" candidate_uuid: "09bbac8eb78a487c9e69c3820ae3cdf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dfa16d7bad9491eb7545fd03602ba25" is_pre_election: true
I20260811 11:50:09.447218 5608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e370aab5a3c4c3f9e6617ca67485a73" candidate_uuid: "09bbac8eb78a487c9e69c3820ae3cdf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a176fbfcbdd440aa4a9fb9fa319262d" is_pre_election: true
I20260811 11:50:09.447620 5385 raft_consensus.cc:2454] T 3e370aab5a3c4c3f9e6617ca67485a73 P 2dfa16d7bad9491eb7545fd03602ba25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09bbac8eb78a487c9e69c3820ae3cdf7 in term 0.
I20260811 11:50:09.448179 5062 leader_election.cc:302] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09bbac8eb78a487c9e69c3820ae3cdf7, 2dfa16d7bad9491eb7545fd03602ba25; no voters:
I20260811 11:50:09.448467 5716 raft_consensus.cc:2781] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:50:09.448550 5716 raft_consensus.cc:482] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:50:09.448585 5716 raft_consensus.cc:3037] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:09.448918 5608 raft_consensus.cc:2454] T 3e370aab5a3c4c3f9e6617ca67485a73 P 8a176fbfcbdd440aa4a9fb9fa319262d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09bbac8eb78a487c9e69c3820ae3cdf7 in term 0.
I20260811 11:50:09.449733 5716 raft_consensus.cc:504] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bbac8eb78a487c9e69c3820ae3cdf7" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41861 } } peers { permanent_uuid: "2dfa16d7bad9491eb7545fd03602ba25" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 35345 } } peers { permanent_uuid: "8a176fbfcbdd440aa4a9fb9fa319262d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 38747 } }
I20260811 11:50:09.450071 5716 leader_election.cc:288] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7 [CANDIDATE]: Term 1 election: Requested vote from peers 2dfa16d7bad9491eb7545fd03602ba25 (127.3.221.66:35345), 8a176fbfcbdd440aa4a9fb9fa319262d (127.3.221.67:38747)
I20260811 11:50:09.450461 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e370aab5a3c4c3f9e6617ca67485a73" candidate_uuid: "09bbac8eb78a487c9e69c3820ae3cdf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dfa16d7bad9491eb7545fd03602ba25"
I20260811 11:50:09.450599 5385 raft_consensus.cc:3037] T 3e370aab5a3c4c3f9e6617ca67485a73 P 2dfa16d7bad9491eb7545fd03602ba25 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:09.450590 5608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e370aab5a3c4c3f9e6617ca67485a73" candidate_uuid: "09bbac8eb78a487c9e69c3820ae3cdf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a176fbfcbdd440aa4a9fb9fa319262d"
I20260811 11:50:09.450735 5608 raft_consensus.cc:3037] T 3e370aab5a3c4c3f9e6617ca67485a73 P 8a176fbfcbdd440aa4a9fb9fa319262d [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:09.451743 5385 raft_consensus.cc:2454] T 3e370aab5a3c4c3f9e6617ca67485a73 P 2dfa16d7bad9491eb7545fd03602ba25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09bbac8eb78a487c9e69c3820ae3cdf7 in term 1.
I20260811 11:50:09.451809 5608 raft_consensus.cc:2454] T 3e370aab5a3c4c3f9e6617ca67485a73 P 8a176fbfcbdd440aa4a9fb9fa319262d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09bbac8eb78a487c9e69c3820ae3cdf7 in term 1.
I20260811 11:50:09.452042 5062 leader_election.cc:302] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09bbac8eb78a487c9e69c3820ae3cdf7, 2dfa16d7bad9491eb7545fd03602ba25; no voters:
I20260811 11:50:09.452323 5716 raft_consensus.cc:2781] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:09.452521 5716 raft_consensus.cc:686] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7 [term 1 LEADER]: Becoming Leader. State: Replica: 09bbac8eb78a487c9e69c3820ae3cdf7, State: Running, Role: LEADER
I20260811 11:50:09.452847 5716 consensus_queue.cc:237] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bbac8eb78a487c9e69c3820ae3cdf7" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41861 } } peers { permanent_uuid: "2dfa16d7bad9491eb7545fd03602ba25" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 35345 } } peers { permanent_uuid: "8a176fbfcbdd440aa4a9fb9fa319262d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 38747 } }
I20260811 11:50:09.456454 4929 catalog_manager.cc:5495] T 3e370aab5a3c4c3f9e6617ca67485a73 P 09bbac8eb78a487c9e69c3820ae3cdf7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09bbac8eb78a487c9e69c3820ae3cdf7 (127.3.221.65). New cstate: current_term: 1 leader_uuid: "09bbac8eb78a487c9e69c3820ae3cdf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bbac8eb78a487c9e69c3820ae3cdf7" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dfa16d7bad9491eb7545fd03602ba25" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 35345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a176fbfcbdd440aa4a9fb9fa319262d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 38747 } health_report { overall_health: UNKNOWN } } }
Aug 11 11:50:09 dist-test-slave-bndk krb5kdc[4849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.65@KRBTEST.COM
I20260811 11:50:09.544262 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:09.535012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58056 (local address 127.3.221.65:41861)
0811 11:50:09.535307 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:09.535310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:09.535342 (+ 32us) server_negotiation.cc:407] Connection header received
0811 11:50:09.535383 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:09.535385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:09.535429 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:09.535518 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:09.536208 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:09.536740 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:09.537683 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:09.537927 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:09.538014 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:09.540661 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:09.540676 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:09.540694 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:09.540721 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:09.542448 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:09.542919 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:09.542926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:09.542927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:09.543014 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:09.543282 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:09.543285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:09.543287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:09.543559 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:09.543774 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:09.543970 (+ 196us) server_negotiation.cc:299] Negotiation successful
0811 11:50:09.544093 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":82,"threads_started":1}
Aug 11 11:50:09 dist-test-slave-bndk krb5kdc[4849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449009, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 11:50:09 dist-test-slave-bndk krb5kdc[4849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449009, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.221.65@KRBTEST.COM
I20260811 11:50:09.573915 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:09.564795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58072 (local address 127.3.221.65:41861)
0811 11:50:09.564992 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:09.564996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:09.565082 (+ 86us) server_negotiation.cc:407] Connection header received
0811 11:50:09.565205 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:09.565208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:09.565288 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:09.565370 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:09.566185 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:09.566706 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:09.567554 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:09.567803 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:09.567890 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:09.570405 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:09.570409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:09.570411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:09.570439 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:09.572190 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:09.572609 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:09.572612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:09.572613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:09.572688 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:09.572994 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:09.572997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:09.572998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:09.573179 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:09.573379 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:09.573638 (+ 259us) server_negotiation.cc:299] Negotiation successful
0811 11:50:09.573754 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
Aug 11 11:50:09 dist-test-slave-bndk krb5kdc[4849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 11:50:09 dist-test-slave-bndk krb5kdc[4849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.65@KRBTEST.COM
I20260811 11:50:09.601127 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:09.592977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58080 (local address 127.3.221.65:41861)
0811 11:50:09.593133 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:09.593137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:09.593338 (+ 201us) server_negotiation.cc:407] Connection header received
0811 11:50:09.593498 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:09.593500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:09.593549 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:09.593615 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:09.594286 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:09.594806 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:09.595657 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:09.595886 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:09.595971 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:09.598487 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:09.598490 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:09.598492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:09.598520 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:09.599686 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:09.600090 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:09.600092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:09.600094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:09.600163 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:09.600425 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:09.600427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:09.600428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:09.600584 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:09.600679 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:09.600903 (+ 224us) server_negotiation.cc:299] Negotiation successful
0811 11:50:09.601001 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260811 11:50:09.602993 5110 tablet_service.cc:2908] Rejecting scan request for tablet 3e370aab5a3c4c3f9e6617ca67485a73: Uninitialized: clean time has not yet been initialized
I20260811 11:50:09.605437 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 5037
I20260811 11:50:09.626710 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 5251
I20260811 11:50:09.643617 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 5466
I20260811 11:50:09.660027 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 4869
2026-08-11T11:50:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3015 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:50:09 dist-test-slave-bndk krb5kdc[5752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:50:09 dist-test-slave-bndk krb5kdc[5752](info): set up 2 sockets
Aug 11 11:50:09 dist-test-slave-bndk krb5kdc[5752](info): commencing operation
krb5kdc: starting...
W20260811 11:50:10.826431 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:50:10 dist-test-slave-bndk krb5kdc[5752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:50:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:50:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:50:10.959733 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:41675
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:41675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:11.076331 5772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:11.076633 5772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:11.076674 5772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:11.080987 5772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:50:11.081076 5772 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 11:50:11.081094 5772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:11.081112 5772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:11.081130 5772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:50:11.081143 5772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:50:11.086825 5772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:41675
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:41675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:11.087954 5772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:11.089017 5772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:11.097827 5779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:11.097838 5786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:11.097857 5782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:11.097908 5772 server_base.cc:1034] running on GCE node
I20260811 11:50:11.098699 5772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:11.099880 5772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:11.101065 5772 hybrid_clock.cc:648] HybridClock initialized: now 1786449011101034 us; error 44 us; skew 500 ppm
Aug 11 11:50:11 dist-test-slave-bndk krb5kdc[5752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449011, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:11.104048 5772 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:50:11.105016 5772 webserver.cc:460] Webserver started at http://127.3.221.126:44029/ using document root <none> and password file <none>
I20260811 11:50:11.105664 5772 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:11.105738 5772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:11.105944 5772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:11.108083 5772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "c15d68e3f93442eca157324cef4b0441"
format_stamp: "Formatted at 2026-08-11 11:50:11 on dist-test-slave-bndk"
server_key: "182c95e7a1e54cc9d1f14082ed06612a"
server_key_iv: "d96a630d25dcf0255a9ef9ed67ec06b6"
server_key_version: "encryptionkey@0"
I20260811 11:50:11.108657 5772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "c15d68e3f93442eca157324cef4b0441"
format_stamp: "Formatted at 2026-08-11 11:50:11 on dist-test-slave-bndk"
server_key: "182c95e7a1e54cc9d1f14082ed06612a"
server_key_iv: "d96a630d25dcf0255a9ef9ed67ec06b6"
server_key_version: "encryptionkey@0"
I20260811 11:50:11.112694 5772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 11:50:11.115357 5797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:11.116694 5772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260811 11:50:11.116837 5772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "c15d68e3f93442eca157324cef4b0441"
format_stamp: "Formatted at 2026-08-11 11:50:11 on dist-test-slave-bndk"
server_key: "182c95e7a1e54cc9d1f14082ed06612a"
server_key_iv: "d96a630d25dcf0255a9ef9ed67ec06b6"
server_key_version: "encryptionkey@0"
I20260811 11:50:11.116948 5772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:11.132237 5772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:11.135259 5772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:11.135466 5772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:11.144690 5772 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:41675
I20260811 11:50:11.144708 5892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:41675 every 8 connection(s)
I20260811 11:50:11.145738 5772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:50:11.147315 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 5772
I20260811 11:50:11.147461 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:50:11.147713 3957 external_mini_cluster.cc:1428] Setting key 3206bfcd8bcf66e3fbdb6aa8c72c4b00
I20260811 11:50:11.148873 5893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:11.155968 5893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441: Bootstrap starting.
Aug 11 11:50:11 dist-test-slave-bndk krb5kdc[5752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:11.158622 5893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:11.160243 5893 log.cc:824] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:11.162778 5893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441: No bootstrap required, opened a new log
I20260811 11:50:11.163694 5898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:11.149187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55908 (local address 127.3.221.126:41675)
0811 11:50:11.149692 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:11.149713 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:11.149759 (+ 46us) server_negotiation.cc:407] Connection header received
0811 11:50:11.150619 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:11.150645 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:11.151073 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:11.151471 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:11.152160 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.153671 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.154567 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.155274 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.155448 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:11.158308 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:11.158314 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:11.158340 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:11.158374 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:11.160419 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:11.160925 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:11.160930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:11.160935 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:11.161056 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:11.161375 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:11.161379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:11.161380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:11.161890 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:11.162085 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:11.162395 (+ 310us) server_negotiation.cc:299] Negotiation successful
0811 11:50:11.162710 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":139,"threads_started":1}
I20260811 11:50:11.166515 5893 raft_consensus.cc:348] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15d68e3f93442eca157324cef4b0441" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 41675 } }
I20260811 11:50:11.166745 5893 raft_consensus.cc:374] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:11.166781 5893 raft_consensus.cc:729] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c15d68e3f93442eca157324cef4b0441, State: Initialized, Role: FOLLOWER
I20260811 11:50:11.167181 5893 consensus_queue.cc:260] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15d68e3f93442eca157324cef4b0441" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 41675 } }
I20260811 11:50:11.167328 5893 raft_consensus.cc:388] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:50:11.167397 5893 raft_consensus.cc:482] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:50:11.167495 5893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:11.168432 5893 raft_consensus.cc:504] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15d68e3f93442eca157324cef4b0441" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 41675 } }
I20260811 11:50:11.168778 5893 leader_election.cc:302] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c15d68e3f93442eca157324cef4b0441; no voters:
I20260811 11:50:11.169092 5893 leader_election.cc:288] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:50:11.169265 5901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:11.169481 5901 raft_consensus.cc:686] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441 [term 1 LEADER]: Becoming Leader. State: Replica: c15d68e3f93442eca157324cef4b0441, State: Running, Role: LEADER
I20260811 11:50:11.169807 5901 consensus_queue.cc:237] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15d68e3f93442eca157324cef4b0441" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 41675 } }
I20260811 11:50:11.170341 5893 sys_catalog.cc:564] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:50:11.172289 5904 sys_catalog.cc:455] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c15d68e3f93442eca157324cef4b0441. Latest consensus state: current_term: 1 leader_uuid: "c15d68e3f93442eca157324cef4b0441" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15d68e3f93442eca157324cef4b0441" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 41675 } } }
I20260811 11:50:11.172434 5904 sys_catalog.cc:458] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:11.172554 5903 sys_catalog.cc:455] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c15d68e3f93442eca157324cef4b0441" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15d68e3f93442eca157324cef4b0441" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 41675 } } }
I20260811 11:50:11.172966 5903 sys_catalog.cc:458] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441 [sys.catalog]: This master's current role is: LEADER
W20260811 11:50:11.175204 5927 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:50:11.175277 5927 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:50:11.175362 5930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:50:11.176072 5930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:50:11.181748 5930 catalog_manager.cc:1347] Generated new cluster ID: 3f4bdf13b09e4c1f9e56661f8551dc94
I20260811 11:50:11.181842 5930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:50:11.189258 5930 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:50:11.190240 5930 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:50:11.197762 5930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441: Generated new TSK 0
I20260811 11:50:11.198475 5930 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
I20260811 11:50:11.237394 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:41675
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:11.370479 5945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:11.370927 5945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:11.370993 5945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:11.375228 5945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:11.375296 5945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:11.375352 5945 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 11:50:11.375419 5945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:50:11.381148 5945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.221.126:41675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:11.382422 5945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:11.383507 5945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:11.393704 5954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:11.393817 5952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:11.393707 5950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:11.393985 5945 server_base.cc:1034] running on GCE node
I20260811 11:50:11.394680 5945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:11.395442 5945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:11.396632 5945 hybrid_clock.cc:648] HybridClock initialized: now 1786449011396611 us; error 34 us; skew 500 ppm
Aug 11 11:50:11 dist-test-slave-bndk krb5kdc[5752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449011, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:11.399629 5945 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:50:11.400614 5945 webserver.cc:460] Webserver started at http://127.3.221.65:44949/ using document root <none> and password file <none>
I20260811 11:50:11.401198 5945 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:11.401316 5945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:11.401551 5945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:11.404068 5945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "3640fd03004d415fa5b95cf3a86c9c48"
format_stamp: "Formatted at 2026-08-11 11:50:11 on dist-test-slave-bndk"
server_key: "4c4236e212842539f39b53ed98214062"
server_key_iv: "96aa49c3b7e17ca7fac822478a0f48a0"
server_key_version: "encryptionkey@0"
I20260811 11:50:11.404654 5945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "3640fd03004d415fa5b95cf3a86c9c48"
format_stamp: "Formatted at 2026-08-11 11:50:11 on dist-test-slave-bndk"
server_key: "4c4236e212842539f39b53ed98214062"
server_key_iv: "96aa49c3b7e17ca7fac822478a0f48a0"
server_key_version: "encryptionkey@0"
I20260811 11:50:11.408594 5945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260811 11:50:11.411361 5967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:11.412703 5945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 11:50:11.412844 5945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "3640fd03004d415fa5b95cf3a86c9c48"
format_stamp: "Formatted at 2026-08-11 11:50:11 on dist-test-slave-bndk"
server_key: "4c4236e212842539f39b53ed98214062"
server_key_iv: "96aa49c3b7e17ca7fac822478a0f48a0"
server_key_version: "encryptionkey@0"
I20260811 11:50:11.412962 5945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:11.443437 5945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:11.446635 5945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:11.446837 5945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:11.448436 5945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:11.448491 5945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:11.448549 5945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:11.448576 5945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 11:50:11 dist-test-slave-bndk krb5kdc[5752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449011, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:11.459267 5945 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:35573
I20260811 11:50:11.459470 6156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:35573 every 8 connection(s)
I20260811 11:50:11.460536 5945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:50:11.466080 5898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:11.449301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:48297 (local address 127.3.221.126:41675)
0811 11:50:11.449458 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:11.449463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:11.450414 (+ 951us) server_negotiation.cc:407] Connection header received
0811 11:50:11.451669 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:11.451672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:11.451737 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:11.451835 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:11.453562 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.454196 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.455478 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.455883 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.456002 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:11.458926 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:11.458930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:11.458933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:11.458965 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:11.461189 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:11.464125 (+ 2936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:11.464130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:11.464132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:11.464245 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:11.464682 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:11.464685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:11.464687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:11.464931 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:11.465030 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:11.465731 (+ 701us) server_negotiation.cc:299] Negotiation successful
0811 11:50:11.465892 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 11:50:11.467331 5978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:11.449703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:41675 (local address 127.3.221.65:48297)
0811 11:50:11.450210 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:11.450280 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:11.451429 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:11.452084 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:11.452095 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:11.452610 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:11.453377 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:11.453399 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.454365 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.454368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:11.455323 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:11.455337 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.456060 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.456065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:11.456300 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:11.456933 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:11.456949 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:11.458738 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:11.463558 (+ 4820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:11.463565 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:11.463581 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:11.463987 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:11.464359 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:11.464363 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:11.464365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:11.464574 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:11.465051 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:11.465059 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:11.465428 (+ 369us) client_negotiation.cc:769] Sending connection context
0811 11:50:11.465651 (+ 223us) client_negotiation.cc:240] Negotiation successful
0811 11:50:11.466065 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":105,"threads_started":1}
I20260811 11:50:11.469012 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 5945
I20260811 11:50:11.469107 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:50:11.469380 3957 external_mini_cluster.cc:1428] Setting key 66681cc838ae0f13d9b179c7b20b6a48
I20260811 11:50:11.473206 6157 heartbeater.cc:347] Connected to a master server at 127.3.221.126:41675
I20260811 11:50:11.473403 6157 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:11.473912 6157 heartbeater.cc:511] Master 127.3.221.126:41675 requested a full tablet report, sending...
I20260811 11:50:11.475510 5821 ts_manager.cc:194] Registered new tserver with Master: 3640fd03004d415fa5b95cf3a86c9c48 (127.3.221.65:35573)
I20260811 11:50:11.476965 5821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:48297
I20260811 11:50:11.481144 5898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:11.473488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:35001 (local address 127.3.221.126:41675)
0811 11:50:11.473614 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:11.473619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:11.474099 (+ 480us) server_negotiation.cc:407] Connection header received
0811 11:50:11.474209 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:11.474211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:11.474262 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:11.474338 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:11.474867 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.475383 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.476271 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.476515 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.476604 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:11.477779 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:11.477783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:11.477786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:11.477815 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:11.479487 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:11.479995 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:11.479999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:11.480001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:11.480084 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:11.480349 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:11.480352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:11.480354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:11.480536 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:11.480623 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:11.480883 (+ 260us) server_negotiation.cc:299] Negotiation successful
0811 11:50:11.480994 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 11:50:11.481158 6165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:11.473506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:41675 (local address 127.3.221.65:35001)
0811 11:50:11.474000 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:11.474012 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:11.474117 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:11.474385 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:11.474388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:11.474629 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:11.474736 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:11.474743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.475518 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.475521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:11.476146 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:11.476156 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.476630 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.476633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:11.476725 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:11.477249 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:11.477266 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:11.477612 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:11.479613 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:11.479617 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:11.479619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:11.479866 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:11.480177 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:11.480180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:11.480181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:11.480251 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:11.480636 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:11.480639 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:11.480720 (+ 81us) client_negotiation.cc:769] Sending connection context
0811 11:50:11.480809 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 11:50:11.480953 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for kudu/127.3.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
I20260811 11:50:11.501408 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:41675
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:11.617420 6173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:11.617746 6173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:11.617812 6173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:11.621973 6173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:11.622049 6173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:11.622105 6173 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 11:50:11.622164 6173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:50:11.627671 6173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.221.126:41675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:11.628940 6173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:11.630004 6173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:11.639670 6188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:11.639655 6186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:11.639657 6183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:11.639916 6173 server_base.cc:1034] running on GCE node
I20260811 11:50:11.640470 6173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:11.641121 6173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:11.642284 6173 hybrid_clock.cc:648] HybridClock initialized: now 1786449011642269 us; error 27 us; skew 500 ppm
Aug 11 11:50:11 dist-test-slave-bndk krb5kdc[5752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449011, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:11.645329 6173 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:50:11.646215 6173 webserver.cc:460] Webserver started at http://127.3.221.66:44305/ using document root <none> and password file <none>
I20260811 11:50:11.646778 6173 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:11.646842 6173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:11.647053 6173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:11.649151 6173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "88db5969cd984867b3e17b6f801fd592"
format_stamp: "Formatted at 2026-08-11 11:50:11 on dist-test-slave-bndk"
server_key: "13361edb554349f1b217f68d4f5d8a85"
server_key_iv: "3dbe4ff3dd63a7c439b4fcfd06de18d8"
server_key_version: "encryptionkey@0"
I20260811 11:50:11.649765 6173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "88db5969cd984867b3e17b6f801fd592"
format_stamp: "Formatted at 2026-08-11 11:50:11 on dist-test-slave-bndk"
server_key: "13361edb554349f1b217f68d4f5d8a85"
server_key_iv: "3dbe4ff3dd63a7c439b4fcfd06de18d8"
server_key_version: "encryptionkey@0"
I20260811 11:50:11.653575 6173 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:50:11.655999 6199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:11.657186 6173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 11:50:11.657347 6173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "88db5969cd984867b3e17b6f801fd592"
format_stamp: "Formatted at 2026-08-11 11:50:11 on dist-test-slave-bndk"
server_key: "13361edb554349f1b217f68d4f5d8a85"
server_key_iv: "3dbe4ff3dd63a7c439b4fcfd06de18d8"
server_key_version: "encryptionkey@0"
I20260811 11:50:11.657449 6173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:11.711970 6173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:11.715322 6173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:11.715526 6173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:11.717114 6173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:11.717168 6173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:11.717247 6173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:11.717281 6173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 11:50:11 dist-test-slave-bndk krb5kdc[5752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449011, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:11.728055 6173 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:42783
I20260811 11:50:11.728101 6392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:42783 every 8 connection(s)
I20260811 11:50:11.729211 6173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:50:11.729427 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 6173
I20260811 11:50:11.729537 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:50:11.729851 3957 external_mini_cluster.cc:1428] Setting key 391c34f17f6963db983ddca76577a0af
I20260811 11:50:11.731523 5898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:11.718091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:60403 (local address 127.3.221.126:41675)
0811 11:50:11.718245 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:11.718258 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:11.719171 (+ 913us) server_negotiation.cc:407] Connection header received
0811 11:50:11.720570 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:11.720574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:11.720641 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:11.720735 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:11.722225 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.723044 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.724333 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.724677 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.724801 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:11.727354 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:11.727358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:11.727361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:11.727394 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:11.729009 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:11.729648 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:11.729652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:11.729654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:11.729752 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:11.730194 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:11.730197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:11.730199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:11.730436 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:11.730528 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:11.731188 (+ 660us) server_negotiation.cc:299] Negotiation successful
0811 11:50:11.731348 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 11:50:11.732306 6217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:11.718503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:41675 (local address 127.3.221.66:60403)
0811 11:50:11.718980 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:11.719057 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:11.720268 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:11.720954 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:11.720964 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:11.721482 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:11.722064 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:11.722079 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.723194 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.723198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:11.724215 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:11.724226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.724857 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.724862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:11.725047 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:11.725626 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:11.725643 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:11.727162 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:11.729141 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:11.729146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:11.729158 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:11.729518 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:11.729871 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:11.729875 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:11.729877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:11.730066 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:11.730543 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:11.730551 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:11.730848 (+ 297us) client_negotiation.cc:769] Sending connection context
0811 11:50:11.731095 (+ 247us) client_negotiation.cc:240] Negotiation successful
0811 11:50:11.731426 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":114,"threads_started":1}
I20260811 11:50:11.734361 6394 heartbeater.cc:347] Connected to a master server at 127.3.221.126:41675
I20260811 11:50:11.734532 6394 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:11.735078 6394 heartbeater.cc:511] Master 127.3.221.126:41675 requested a full tablet report, sending...
I20260811 11:50:11.736282 5822 ts_manager.cc:194] Registered new tserver with Master: 88db5969cd984867b3e17b6f801fd592 (127.3.221.66:42783)
I20260811 11:50:11.737017 5822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:60403
WARNING: no policy specified for kudu/127.3.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.67@KRBTEST.COM" created.
I20260811 11:50:11.748246 5898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:11.736864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:55543 (local address 127.3.221.126:41675)
0811 11:50:11.736969 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:11.736973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:11.741619 (+ 4646us) server_negotiation.cc:407] Connection header received
0811 11:50:11.741746 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:11.741749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:11.741805 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:11.741872 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:11.742360 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.742885 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.743691 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.743923 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.744009 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:11.745098 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:11.745102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:11.745105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:11.745130 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:11.746612 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:11.747049 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:11.747052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:11.747054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:11.747145 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:11.747375 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:11.747378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:11.747379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:11.747653 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:11.747745 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:11.748015 (+ 270us) server_negotiation.cc:299] Negotiation successful
0811 11:50:11.748119 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 11:50:11.748312 6409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:11.736784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:41675 (local address 127.3.221.66:55543)
0811 11:50:11.741506 (+ 4722us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:11.741520 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:11.741650 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:11.741915 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:11.741918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:11.742125 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:11.742232 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:11.742238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.743015 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.743018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:11.743576 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:11.743584 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.744037 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.744041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:11.744135 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:11.744622 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:11.744637 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:11.744959 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:11.746735 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:11.746738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:11.746740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:11.746947 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:11.747225 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:11.747227 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:11.747228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:11.747299 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:11.747755 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:11.747758 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:11.747859 (+ 101us) client_negotiation.cc:769] Sending connection context
0811 11:50:11.747968 (+ 109us) client_negotiation.cc:240] Negotiation successful
0811 11:50:11.748116 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4619,"thread_start_us":141,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
I20260811 11:50:11.762096 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:41675
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:11.878806 6415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:11.879088 6415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:11.879133 6415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:11.883335 6415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:11.883416 6415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:11.883455 6415 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 11:50:11.883489 6415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:50:11.888948 6415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.221.126:41675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:11.890116 6415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:11.891201 6415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:11.901326 6423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:11.901448 6415 server_base.cc:1034] running on GCE node
W20260811 11:50:11.901351 6422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:11.901319 6425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:11.902052 6415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:11.902729 6415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:11.903925 6415 hybrid_clock.cc:648] HybridClock initialized: now 1786449011903906 us; error 30 us; skew 500 ppm
Aug 11 11:50:11 dist-test-slave-bndk krb5kdc[5752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449011, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:11.906893 6415 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:50:11.907855 6415 webserver.cc:460] Webserver started at http://127.3.221.67:39423/ using document root <none> and password file <none>
I20260811 11:50:11.908416 6415 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:11.908505 6415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:11.908730 6415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:11.910880 6415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8"
format_stamp: "Formatted at 2026-08-11 11:50:11 on dist-test-slave-bndk"
server_key: "aed19ad4db69d46a453fe3c71ceade28"
server_key_iv: "94d591b40f93c9c480112a985634544d"
server_key_version: "encryptionkey@0"
I20260811 11:50:11.911489 6415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8"
format_stamp: "Formatted at 2026-08-11 11:50:11 on dist-test-slave-bndk"
server_key: "aed19ad4db69d46a453fe3c71ceade28"
server_key_iv: "94d591b40f93c9c480112a985634544d"
server_key_version: "encryptionkey@0"
I20260811 11:50:11.915452 6415 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 11:50:11.918058 6433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:11.919327 6415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 11:50:11.919445 6415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8"
format_stamp: "Formatted at 2026-08-11 11:50:11 on dist-test-slave-bndk"
server_key: "aed19ad4db69d46a453fe3c71ceade28"
server_key_iv: "94d591b40f93c9c480112a985634544d"
server_key_version: "encryptionkey@0"
I20260811 11:50:11.919564 6415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:11.945174 6415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:11.948312 6415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:11.948518 6415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:11.950112 6415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:11.950167 6415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:11.950223 6415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:11.950251 6415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 11:50:11 dist-test-slave-bndk krb5kdc[5752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449011, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:11.961015 6415 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:37089
I20260811 11:50:11.961191 6612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:37089 every 8 connection(s)
I20260811 11:50:11.962251 6415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:50:11.964443 5898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:11.950929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:52729 (local address 127.3.221.126:41675)
0811 11:50:11.951105 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:11.951110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:11.952046 (+ 936us) server_negotiation.cc:407] Connection header received
0811 11:50:11.953114 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:11.953118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:11.953186 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:11.953313 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:11.954670 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.955484 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.956747 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.957104 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.957266 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:11.960196 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:11.960200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:11.960203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:11.960234 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:11.962044 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:11.962671 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:11.962676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:11.962677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:11.962797 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:11.963250 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:11.963254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:11.963256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:11.963476 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:11.963574 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:11.964128 (+ 554us) server_negotiation.cc:299] Negotiation successful
0811 11:50:11.964270 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 11:50:11.965353 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:11.951373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:41675 (local address 127.3.221.67:52729)
0811 11:50:11.951863 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:11.951916 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:11.952861 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:11.953501 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:11.953509 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:11.953973 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:11.954524 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:11.954538 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.955640 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.955644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:11.956612 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:11.956624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.957327 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.957331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:11.957520 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:11.958155 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:11.958177 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:11.960004 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:11.962184 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:11.962191 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:11.962204 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:11.962539 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:11.962915 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:11.962919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:11.962921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:11.963134 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:11.963594 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:11.963601 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:11.963866 (+ 265us) client_negotiation.cc:769] Sending connection context
0811 11:50:11.964050 (+ 184us) client_negotiation.cc:240] Negotiation successful
0811 11:50:11.964398 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":114,"threads_started":1}
I20260811 11:50:11.966673 6613 heartbeater.cc:347] Connected to a master server at 127.3.221.126:41675
I20260811 11:50:11.966826 6613 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:11.967360 6613 heartbeater.cc:511] Master 127.3.221.126:41675 requested a full tablet report, sending...
I20260811 11:50:11.968516 5822 ts_manager.cc:194] Registered new tserver with Master: 1e2f5b88af5f4378ad32ba9cfc7499f8 (127.3.221.67:37089)
I20260811 11:50:11.969310 5822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:52729
I20260811 11:50:11.969846 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 6415
I20260811 11:50:11.969966 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:50:11.970269 3957 external_mini_cluster.cc:1428] Setting key 84fbb0fef143fe406f15c9ed36c0f402
I20260811 11:50:11.972283 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:50:11.976774 5898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:11.968992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:34545 (local address 127.3.221.126:41675)
0811 11:50:11.969110 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:11.969115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:11.969331 (+ 216us) server_negotiation.cc:407] Connection header received
0811 11:50:11.969476 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:11.969478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:11.969517 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:11.969594 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:11.970112 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.971024 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.972134 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.972469 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.972551 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:11.973772 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:11.973775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:11.973777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:11.973801 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:11.975223 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:11.975629 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:11.975633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:11.975634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:11.975728 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:11.975976 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:11.975979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:11.975981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:11.976198 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:11.976294 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:11.976556 (+ 262us) server_negotiation.cc:299] Negotiation successful
0811 11:50:11.976657 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 11:50:11.976818 6626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:11.968927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:41675 (local address 127.3.221.67:34545)
0811 11:50:11.969243 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:11.969258 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:11.969368 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:11.969631 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:11.969634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:11.969851 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:11.969999 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:11.970007 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.971188 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.971192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:11.971995 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:11.972004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.972581 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.972585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:11.972684 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:11.973194 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:11.973208 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:11.973640 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:11.975336 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:11.975339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:11.975340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:11.975538 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:11.975813 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:11.975815 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:11.975816 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:11.975898 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:11.976301 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:11.976304 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:11.976406 (+ 102us) client_negotiation.cc:769] Sending connection context
0811 11:50:11.976507 (+ 101us) client_negotiation.cc:240] Negotiation successful
0811 11:50:11.976651 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":1,"thread_start_us":112,"threads_started":1}
Aug 11 11:50:11 dist-test-slave-bndk krb5kdc[5752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449011, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 11:50:11 dist-test-slave-bndk krb5kdc[5752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449011, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:11.999642 5898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:11.991983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55918 (local address 127.3.221.126:41675)
0811 11:50:11.992113 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:11.992116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:11.992130 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:50:11.992175 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:11.992178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:11.992220 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:11.992290 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:11.992981 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.993549 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.994351 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:11.994569 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:11.994637 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:11.997005 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:11.997008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:11.997010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:11.997039 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:11.998288 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:11.998691 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:11.998693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:11.998695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:11.998765 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:11.998955 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:11.998957 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:11.998958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:11.999131 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:11.999234 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:11.999410 (+ 176us) server_negotiation.cc:299] Negotiation successful
0811 11:50:11.999514 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 11:50:12.002907 5822 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:50:12.006054 5822 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:50:12.020536 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.015055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:42783 (local address 127.0.0.1:52954)
0811 11:50:12.015971 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:12.015985 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:12.016077 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:12.016579 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:12.016583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:12.016602 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:12.016747 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.016754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.017880 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.017883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.019106 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.019117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.019972 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.019976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.020086 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.020126 (+ 40us) client_negotiation.cc:769] Sending connection context
0811 11:50:12.020236 (+ 110us) client_negotiation.cc:240] Negotiation successful
0811 11:50:12.020360 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"mutex_wait_us":70,"thread_start_us":115,"threads_started":1}
I20260811 11:50:12.020569 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.014726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:35573 (local address 127.0.0.1:41048)
0811 11:50:12.015583 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:12.015637 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:12.015777 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:12.016484 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:12.016491 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:12.016518 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:12.016717 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.016726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.017816 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.017820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.019206 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.019216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.020252 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.020255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.020337 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.020358 (+ 21us) client_negotiation.cc:769] Sending connection context
0811 11:50:12.020413 (+ 55us) client_negotiation.cc:240] Negotiation successful
0811 11:50:12.020474 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":69,"thread_start_us":106,"threads_started":1}
I20260811 11:50:12.020980 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.014883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41048 (local address 127.3.221.65:35573)
0811 11:50:12.015691 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:12.015698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:12.015724 (+ 26us) server_negotiation.cc:407] Connection header received
0811 11:50:12.016013 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:12.016019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:12.016303 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:12.016480 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:12.016847 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.017705 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.019333 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.020148 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.020252 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.020396 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:12.020498 (+ 102us) server_negotiation.cc:299] Negotiation successful
0811 11:50:12.020604 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":92,"threads_started":1}
I20260811 11:50:12.021024 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.015238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.3.221.66:42783)
0811 11:50:12.016114 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:12.016126 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:12.016148 (+ 22us) server_negotiation.cc:407] Connection header received
0811 11:50:12.016244 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:12.016251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:12.016445 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:12.016598 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:12.016880 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.017762 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.019242 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.019845 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.019951 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.020144 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:12.020780 (+ 636us) server_negotiation.cc:299] Negotiation successful
0811 11:50:12.020879 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":131,"threads_started":1}
I20260811 11:50:12.023779 6270 tablet_service.cc:1467] Processing CreateTablet for tablet e90e4c0458f94776ae641cba8b4e6c3c (DEFAULT_TABLE table=test-table [id=f231748cdd7b4fde901dc3dfc7f4f6a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:12.024214 6645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.015509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:37089 (local address 127.0.0.1:47798)
0811 11:50:12.016712 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:12.016725 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:12.016834 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:12.018855 (+ 2021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:12.018858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:12.018875 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:12.018985 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.018991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.020429 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.020433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.022351 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.022363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.023737 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.023741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.023839 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.023864 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 11:50:12.023939 (+ 75us) client_negotiation.cc:240] Negotiation successful
0811 11:50:12.024016 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1147,"thread_start_us":95,"threads_started":1}
I20260811 11:50:12.023927 6045 tablet_service.cc:1467] Processing CreateTablet for tablet e90e4c0458f94776ae641cba8b4e6c3c (DEFAULT_TABLE table=test-table [id=f231748cdd7b4fde901dc3dfc7f4f6a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:12.024715 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.015956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47798 (local address 127.3.221.67:37089)
0811 11:50:12.016464 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:12.016479 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:12.017828 (+ 1349us) server_negotiation.cc:407] Connection header received
0811 11:50:12.017940 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:12.017946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:12.018557 (+ 611us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:12.018712 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:12.019115 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.020301 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.022502 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.023408 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.023511 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.023670 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:12.024470 (+ 800us) server_negotiation.cc:299] Negotiation successful
0811 11:50:12.024574 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":148,"threads_started":1}
I20260811 11:50:12.024940 6270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e90e4c0458f94776ae641cba8b4e6c3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:12.025027 6045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e90e4c0458f94776ae641cba8b4e6c3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:12.027565 6518 tablet_service.cc:1467] Processing CreateTablet for tablet e90e4c0458f94776ae641cba8b4e6c3c (DEFAULT_TABLE table=test-table [id=f231748cdd7b4fde901dc3dfc7f4f6a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:12.028733 6518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e90e4c0458f94776ae641cba8b4e6c3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:12.032431 6647 tablet_bootstrap.cc:492] T e90e4c0458f94776ae641cba8b4e6c3c P 88db5969cd984867b3e17b6f801fd592: Bootstrap starting.
I20260811 11:50:12.034174 6648 tablet_bootstrap.cc:492] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48: Bootstrap starting.
I20260811 11:50:12.035494 6647 tablet_bootstrap.cc:654] T e90e4c0458f94776ae641cba8b4e6c3c P 88db5969cd984867b3e17b6f801fd592: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:12.036285 6647 log.cc:824] T e90e4c0458f94776ae641cba8b4e6c3c P 88db5969cd984867b3e17b6f801fd592: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:12.036513 6648 tablet_bootstrap.cc:654] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:12.037465 6649 tablet_bootstrap.cc:492] T e90e4c0458f94776ae641cba8b4e6c3c P 1e2f5b88af5f4378ad32ba9cfc7499f8: Bootstrap starting.
I20260811 11:50:12.037616 6648 log.cc:824] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:12.038592 6647 tablet_bootstrap.cc:492] T e90e4c0458f94776ae641cba8b4e6c3c P 88db5969cd984867b3e17b6f801fd592: No bootstrap required, opened a new log
I20260811 11:50:12.038791 6647 ts_tablet_manager.cc:1397] T e90e4c0458f94776ae641cba8b4e6c3c P 88db5969cd984867b3e17b6f801fd592: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260811 11:50:12.039608 6648 tablet_bootstrap.cc:492] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48: No bootstrap required, opened a new log
I20260811 11:50:12.039739 6649 tablet_bootstrap.cc:654] T e90e4c0458f94776ae641cba8b4e6c3c P 1e2f5b88af5f4378ad32ba9cfc7499f8: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:12.039760 6648 ts_tablet_manager.cc:1397] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 11:50:12.040592 6649 log.cc:824] T e90e4c0458f94776ae641cba8b4e6c3c P 1e2f5b88af5f4378ad32ba9cfc7499f8: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:12.042888 6649 tablet_bootstrap.cc:492] T e90e4c0458f94776ae641cba8b4e6c3c P 1e2f5b88af5f4378ad32ba9cfc7499f8: No bootstrap required, opened a new log
I20260811 11:50:12.042793 6647 raft_consensus.cc:348] T e90e4c0458f94776ae641cba8b4e6c3c P 88db5969cd984867b3e17b6f801fd592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } }
I20260811 11:50:12.043102 6649 ts_tablet_manager.cc:1397] T e90e4c0458f94776ae641cba8b4e6c3c P 1e2f5b88af5f4378ad32ba9cfc7499f8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 11:50:12.043098 6647 raft_consensus.cc:374] T e90e4c0458f94776ae641cba8b4e6c3c P 88db5969cd984867b3e17b6f801fd592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:12.043159 6647 raft_consensus.cc:729] T e90e4c0458f94776ae641cba8b4e6c3c P 88db5969cd984867b3e17b6f801fd592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88db5969cd984867b3e17b6f801fd592, State: Initialized, Role: FOLLOWER
I20260811 11:50:12.043347 6648 raft_consensus.cc:348] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } }
I20260811 11:50:12.043612 6648 raft_consensus.cc:374] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:12.043669 6648 raft_consensus.cc:729] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3640fd03004d415fa5b95cf3a86c9c48, State: Initialized, Role: FOLLOWER
I20260811 11:50:12.043678 6647 consensus_queue.cc:260] T e90e4c0458f94776ae641cba8b4e6c3c P 88db5969cd984867b3e17b6f801fd592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } }
I20260811 11:50:12.044119 6648 consensus_queue.cc:260] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } }
I20260811 11:50:12.044541 6394 heartbeater.cc:503] Master 127.3.221.126:41675 was elected leader, sending a full tablet report...
I20260811 11:50:12.044831 6647 ts_tablet_manager.cc:1428] T e90e4c0458f94776ae641cba8b4e6c3c P 88db5969cd984867b3e17b6f801fd592: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 11:50:12.044973 6648 ts_tablet_manager.cc:1428] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260811 11:50:12.045374 6157 heartbeater.cc:503] Master 127.3.221.126:41675 was elected leader, sending a full tablet report...
I20260811 11:50:12.048261 6649 raft_consensus.cc:348] T e90e4c0458f94776ae641cba8b4e6c3c P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } }
I20260811 11:50:12.048501 6649 raft_consensus.cc:374] T e90e4c0458f94776ae641cba8b4e6c3c P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:12.048563 6649 raft_consensus.cc:729] T e90e4c0458f94776ae641cba8b4e6c3c P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2f5b88af5f4378ad32ba9cfc7499f8, State: Initialized, Role: FOLLOWER
I20260811 11:50:12.049037 6649 consensus_queue.cc:260] T e90e4c0458f94776ae641cba8b4e6c3c P 1e2f5b88af5f4378ad32ba9cfc7499f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } }
I20260811 11:50:12.049782 6613 heartbeater.cc:503] Master 127.3.221.126:41675 was elected leader, sending a full tablet report...
I20260811 11:50:12.050173 6649 ts_tablet_manager.cc:1428] T e90e4c0458f94776ae641cba8b4e6c3c P 1e2f5b88af5f4378ad32ba9cfc7499f8: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260811 11:50:12.120609 6660 raft_consensus.cc:482] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:50:12.120846 6660 raft_consensus.cc:504] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } }
I20260811 11:50:12.122218 6660 leader_election.cc:288] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2f5b88af5f4378ad32ba9cfc7499f8 (127.3.221.67:37089), 88db5969cd984867b3e17b6f801fd592 (127.3.221.66:42783)
I20260811 11:50:12.127671 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.122654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:57985 (local address 127.3.221.66:42783)
0811 11:50:12.122792 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:12.122797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:12.123060 (+ 263us) server_negotiation.cc:407] Connection header received
0811 11:50:12.123169 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:12.123172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:12.123224 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:12.123307 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:12.123761 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.124547 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.126368 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.126992 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.127123 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.127180 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:12.127453 (+ 273us) server_negotiation.cc:299] Negotiation successful
0811 11:50:12.127535 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 11:50:12.127671 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.122570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:41851 (local address 127.3.221.67:37089)
0811 11:50:12.122736 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:12.122740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:12.123037 (+ 297us) server_negotiation.cc:407] Connection header received
0811 11:50:12.123173 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:12.123175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:12.123224 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:12.123307 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:12.123716 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.124542 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.126371 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.127003 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.127123 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.127179 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:12.127453 (+ 274us) server_negotiation.cc:299] Negotiation successful
0811 11:50:12.127531 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 11:50:12.127731 6664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.122512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:42783 (local address 127.3.221.65:57985)
0811 11:50:12.122945 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:12.122958 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:12.123073 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:12.123333 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:12.123336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:12.123377 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:12.123595 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.123604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.124702 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.124706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.126233 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.126247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.127152 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.127156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.127295 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.127342 (+ 47us) client_negotiation.cc:769] Sending connection context
0811 11:50:12.127429 (+ 87us) client_negotiation.cc:240] Negotiation successful
0811 11:50:12.127527 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":153,"spinlock_wait_cycles":79360,"thread_start_us":137,"threads_started":1}
I20260811 11:50:12.127731 6665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.122429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:37089 (local address 127.3.221.65:41851)
0811 11:50:12.122888 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:12.122902 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:12.123073 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:12.123333 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:12.123336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:12.123379 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:12.123595 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.123604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.124702 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.124706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.126233 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.126247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.127152 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.127156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.127290 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.127333 (+ 43us) client_negotiation.cc:769] Sending connection context
0811 11:50:12.127429 (+ 96us) client_negotiation.cc:240] Negotiation successful
0811 11:50:12.127528 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":188,"spinlock_wait_cycles":10240,"thread_start_us":173,"threads_started":1}
I20260811 11:50:12.128453 6321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90e4c0458f94776ae641cba8b4e6c3c" candidate_uuid: "3640fd03004d415fa5b95cf3a86c9c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88db5969cd984867b3e17b6f801fd592" is_pre_election: true
I20260811 11:50:12.128618 6548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90e4c0458f94776ae641cba8b4e6c3c" candidate_uuid: "3640fd03004d415fa5b95cf3a86c9c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" is_pre_election: true
I20260811 11:50:12.128849 6548 raft_consensus.cc:2454] T e90e4c0458f94776ae641cba8b4e6c3c P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3640fd03004d415fa5b95cf3a86c9c48 in term 0.
I20260811 11:50:12.128834 6321 raft_consensus.cc:2454] T e90e4c0458f94776ae641cba8b4e6c3c P 88db5969cd984867b3e17b6f801fd592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3640fd03004d415fa5b95cf3a86c9c48 in term 0.
I20260811 11:50:12.129419 5973 leader_election.cc:302] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2f5b88af5f4378ad32ba9cfc7499f8, 3640fd03004d415fa5b95cf3a86c9c48; no voters:
I20260811 11:50:12.129738 6660 raft_consensus.cc:2781] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:50:12.129830 6660 raft_consensus.cc:482] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:50:12.129895 6660 raft_consensus.cc:3037] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:12.130990 6660 raft_consensus.cc:504] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } }
I20260811 11:50:12.131364 6660 leader_election.cc:288] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48 [CANDIDATE]: Term 1 election: Requested vote from peers 1e2f5b88af5f4378ad32ba9cfc7499f8 (127.3.221.67:37089), 88db5969cd984867b3e17b6f801fd592 (127.3.221.66:42783)
I20260811 11:50:12.131733 6548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90e4c0458f94776ae641cba8b4e6c3c" candidate_uuid: "3640fd03004d415fa5b95cf3a86c9c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8"
I20260811 11:50:12.131867 6548 raft_consensus.cc:3037] T e90e4c0458f94776ae641cba8b4e6c3c P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:12.132884 6548 raft_consensus.cc:2454] T e90e4c0458f94776ae641cba8b4e6c3c P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3640fd03004d415fa5b95cf3a86c9c48 in term 1.
I20260811 11:50:12.133155 6321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90e4c0458f94776ae641cba8b4e6c3c" candidate_uuid: "3640fd03004d415fa5b95cf3a86c9c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88db5969cd984867b3e17b6f801fd592"
I20260811 11:50:12.133244 5973 leader_election.cc:302] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2f5b88af5f4378ad32ba9cfc7499f8, 3640fd03004d415fa5b95cf3a86c9c48; no voters:
I20260811 11:50:12.133315 6321 raft_consensus.cc:3037] T e90e4c0458f94776ae641cba8b4e6c3c P 88db5969cd984867b3e17b6f801fd592 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:12.133440 6660 raft_consensus.cc:2781] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:12.133687 6660 raft_consensus.cc:686] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48 [term 1 LEADER]: Becoming Leader. State: Replica: 3640fd03004d415fa5b95cf3a86c9c48, State: Running, Role: LEADER
I20260811 11:50:12.134040 6660 consensus_queue.cc:237] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } }
I20260811 11:50:12.134971 6321 raft_consensus.cc:2454] T e90e4c0458f94776ae641cba8b4e6c3c P 88db5969cd984867b3e17b6f801fd592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3640fd03004d415fa5b95cf3a86c9c48 in term 1.
I20260811 11:50:12.137668 5822 catalog_manager.cc:5495] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3640fd03004d415fa5b95cf3a86c9c48 (127.3.221.65). New cstate: current_term: 1 leader_uuid: "3640fd03004d415fa5b95cf3a86c9c48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } health_report { overall_health: UNKNOWN } } }
I20260811 11:50:12.184381 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.180655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41052 (local address 127.3.221.65:35573)
0811 11:50:12.180799 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:12.180803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:12.180818 (+ 15us) server_negotiation.cc:407] Connection header received
0811 11:50:12.181011 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:12.181013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:12.181068 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:12.181152 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:12.181476 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.181989 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.182916 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.183130 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.183207 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.183499 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:12.183958 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:12.184050 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:12.184171 (+ 121us) server_negotiation.cc:299] Negotiation successful
0811 11:50:12.184225 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 11:50:12.191417 6321 raft_consensus.cc:1261] T e90e4c0458f94776ae641cba8b4e6c3c P 88db5969cd984867b3e17b6f801fd592 [term 1 FOLLOWER]: Refusing update from remote peer 3640fd03004d415fa5b95cf3a86c9c48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:12.192265 6666 consensus_queue.cc:1035] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:12.192643 6548 raft_consensus.cc:1261] T e90e4c0458f94776ae641cba8b4e6c3c P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 1 FOLLOWER]: Refusing update from remote peer 3640fd03004d415fa5b95cf3a86c9c48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:12.193279 6666 consensus_queue.cc:1035] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:12.202462 6675 mvcc.cc:205] Tried to move back new op lower bound from 7317295153925558272 to 7317295153703890944. Current Snapshot: MvccSnapshot[applied={T|T < 7317295153925558272}]
I20260811 11:50:12.215811 5821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:12.216115 5821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:12.219113 5821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441: Sending DeleteTablet for 3 replicas of tablet e90e4c0458f94776ae641cba8b4e6c3c
I20260811 11:50:12.220170 6518 tablet_service.cc:1514] Processing DeleteTablet for tablet e90e4c0458f94776ae641cba8b4e6c3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:12 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:47798
I20260811 11:50:12.220192 6270 tablet_service.cc:1514] Processing DeleteTablet for tablet e90e4c0458f94776ae641cba8b4e6c3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:12 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:52954
I20260811 11:50:12.220698 6684 tablet_replica.cc:330] T e90e4c0458f94776ae641cba8b4e6c3c P 1e2f5b88af5f4378ad32ba9cfc7499f8: stopping tablet replica
I20260811 11:50:12.220937 6045 tablet_service.cc:1514] Processing DeleteTablet for tablet e90e4c0458f94776ae641cba8b4e6c3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:12 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:41048
I20260811 11:50:12.221143 6684 raft_consensus.cc:2229] T e90e4c0458f94776ae641cba8b4e6c3c P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:12.221478 6684 raft_consensus.cc:2258] T e90e4c0458f94776ae641cba8b4e6c3c P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:12.222030 6686 tablet_replica.cc:330] T e90e4c0458f94776ae641cba8b4e6c3c P 88db5969cd984867b3e17b6f801fd592: stopping tablet replica
I20260811 11:50:12.222338 6686 raft_consensus.cc:2229] T e90e4c0458f94776ae641cba8b4e6c3c P 88db5969cd984867b3e17b6f801fd592 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:12.222278 5821 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260811 11:50:12.222640 6686 raft_consensus.cc:2258] T e90e4c0458f94776ae641cba8b4e6c3c P 88db5969cd984867b3e17b6f801fd592 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260811 11:50:12.222687 5821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:50:12.223013 6689 tablet_replica.cc:330] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48: stopping tablet replica
I20260811 11:50:12.223346 6689 raft_consensus.cc:2229] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:50:12.223742 6689 raft_consensus.cc:2258] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:12.225205 6689 ts_tablet_manager.cc:1905] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:12.225205 6684 ts_tablet_manager.cc:1905] T e90e4c0458f94776ae641cba8b4e6c3c P 1e2f5b88af5f4378ad32ba9cfc7499f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:12.228147 6689 ts_tablet_manager.cc:1918] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:50:12.228158 6684 ts_tablet_manager.cc:1918] T e90e4c0458f94776ae641cba8b4e6c3c P 1e2f5b88af5f4378ad32ba9cfc7499f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:50:12.228257 6689 log.cc:1196] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-0/wal/wals/e90e4c0458f94776ae641cba8b4e6c3c
I20260811 11:50:12.228264 6684 log.cc:1196] T e90e4c0458f94776ae641cba8b4e6c3c P 1e2f5b88af5f4378ad32ba9cfc7499f8: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/e90e4c0458f94776ae641cba8b4e6c3c
I20260811 11:50:12.228608 6689 ts_tablet_manager.cc:1939] T e90e4c0458f94776ae641cba8b4e6c3c P 3640fd03004d415fa5b95cf3a86c9c48: Deleting consensus metadata
I20260811 11:50:12.228612 6684 ts_tablet_manager.cc:1939] T e90e4c0458f94776ae641cba8b4e6c3c P 1e2f5b88af5f4378ad32ba9cfc7499f8: Deleting consensus metadata
I20260811 11:50:12.228834 6045 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7e807a584449b484cc23551946861b (DEFAULT_TABLE table=test-table [id=8488453205514b238f26a4472d47029d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:12.229096 6045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7e807a584449b484cc23551946861b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:12.229848 5802 catalog_manager.cc:4841] TS 3640fd03004d415fa5b95cf3a86c9c48 (127.3.221.65:35573): tablet e90e4c0458f94776ae641cba8b4e6c3c (table test-table [id=f231748cdd7b4fde901dc3dfc7f4f6a2]) successfully deleted
I20260811 11:50:12.231124 6648 tablet_bootstrap.cc:492] T 3c7e807a584449b484cc23551946861b P 3640fd03004d415fa5b95cf3a86c9c48: Bootstrap starting.
I20260811 11:50:12.231383 6518 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7e807a584449b484cc23551946861b (DEFAULT_TABLE table=test-table [id=8488453205514b238f26a4472d47029d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:12.231630 6518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7e807a584449b484cc23551946861b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:12.231657 6270 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7e807a584449b484cc23551946861b (DEFAULT_TABLE table=test-table [id=8488453205514b238f26a4472d47029d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:12.231933 6270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7e807a584449b484cc23551946861b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:12.232136 6648 tablet_bootstrap.cc:654] T 3c7e807a584449b484cc23551946861b P 3640fd03004d415fa5b95cf3a86c9c48: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:12.233806 6649 tablet_bootstrap.cc:492] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8: Bootstrap starting.
I20260811 11:50:12.233824 5802 catalog_manager.cc:4841] TS 1e2f5b88af5f4378ad32ba9cfc7499f8 (127.3.221.67:37089): tablet e90e4c0458f94776ae641cba8b4e6c3c (table test-table [id=f231748cdd7b4fde901dc3dfc7f4f6a2]) successfully deleted
I20260811 11:50:12.234455 6647 tablet_bootstrap.cc:492] T 3c7e807a584449b484cc23551946861b P 88db5969cd984867b3e17b6f801fd592: Bootstrap starting.
I20260811 11:50:12.234591 6648 tablet_bootstrap.cc:492] T 3c7e807a584449b484cc23551946861b P 3640fd03004d415fa5b95cf3a86c9c48: No bootstrap required, opened a new log
I20260811 11:50:12.234676 6648 ts_tablet_manager.cc:1397] T 3c7e807a584449b484cc23551946861b P 3640fd03004d415fa5b95cf3a86c9c48: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260811 11:50:12.234966 6649 tablet_bootstrap.cc:654] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:12.235190 6648 raft_consensus.cc:348] T 3c7e807a584449b484cc23551946861b P 3640fd03004d415fa5b95cf3a86c9c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } }
I20260811 11:50:12.235323 6648 raft_consensus.cc:374] T 3c7e807a584449b484cc23551946861b P 3640fd03004d415fa5b95cf3a86c9c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:12.235373 6648 raft_consensus.cc:729] T 3c7e807a584449b484cc23551946861b P 3640fd03004d415fa5b95cf3a86c9c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3640fd03004d415fa5b95cf3a86c9c48, State: Initialized, Role: FOLLOWER
I20260811 11:50:12.235396 6647 tablet_bootstrap.cc:654] T 3c7e807a584449b484cc23551946861b P 88db5969cd984867b3e17b6f801fd592: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:12.235496 6648 consensus_queue.cc:260] T 3c7e807a584449b484cc23551946861b P 3640fd03004d415fa5b95cf3a86c9c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } }
I20260811 11:50:12.235800 6686 ts_tablet_manager.cc:1905] T e90e4c0458f94776ae641cba8b4e6c3c P 88db5969cd984867b3e17b6f801fd592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:12.236539 6648 ts_tablet_manager.cc:1428] T 3c7e807a584449b484cc23551946861b P 3640fd03004d415fa5b95cf3a86c9c48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:50:12.238143 6647 tablet_bootstrap.cc:492] T 3c7e807a584449b484cc23551946861b P 88db5969cd984867b3e17b6f801fd592: No bootstrap required, opened a new log
I20260811 11:50:12.238238 6647 ts_tablet_manager.cc:1397] T 3c7e807a584449b484cc23551946861b P 88db5969cd984867b3e17b6f801fd592: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260811 11:50:12.238770 6686 ts_tablet_manager.cc:1918] T e90e4c0458f94776ae641cba8b4e6c3c P 88db5969cd984867b3e17b6f801fd592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:50:12.238955 6686 log.cc:1196] T e90e4c0458f94776ae641cba8b4e6c3c P 88db5969cd984867b3e17b6f801fd592: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/e90e4c0458f94776ae641cba8b4e6c3c
I20260811 11:50:12.239338 6686 ts_tablet_manager.cc:1939] T e90e4c0458f94776ae641cba8b4e6c3c P 88db5969cd984867b3e17b6f801fd592: Deleting consensus metadata
I20260811 11:50:12.238734 6647 raft_consensus.cc:348] T 3c7e807a584449b484cc23551946861b P 88db5969cd984867b3e17b6f801fd592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } }
I20260811 11:50:12.239542 6647 raft_consensus.cc:374] T 3c7e807a584449b484cc23551946861b P 88db5969cd984867b3e17b6f801fd592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:12.239593 6647 raft_consensus.cc:729] T 3c7e807a584449b484cc23551946861b P 88db5969cd984867b3e17b6f801fd592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88db5969cd984867b3e17b6f801fd592, State: Initialized, Role: FOLLOWER
I20260811 11:50:12.239729 6647 consensus_queue.cc:260] T 3c7e807a584449b484cc23551946861b P 88db5969cd984867b3e17b6f801fd592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } }
I20260811 11:50:12.240025 6647 ts_tablet_manager.cc:1428] T 3c7e807a584449b484cc23551946861b P 88db5969cd984867b3e17b6f801fd592: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 11:50:12.240605 5800 catalog_manager.cc:4841] TS 88db5969cd984867b3e17b6f801fd592 (127.3.221.66:42783): tablet e90e4c0458f94776ae641cba8b4e6c3c (table test-table [id=f231748cdd7b4fde901dc3dfc7f4f6a2]) successfully deleted
I20260811 11:50:12.242031 6649 tablet_bootstrap.cc:492] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8: No bootstrap required, opened a new log
I20260811 11:50:12.242117 6649 ts_tablet_manager.cc:1397] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260811 11:50:12.242580 6649 raft_consensus.cc:348] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } }
I20260811 11:50:12.242687 6649 raft_consensus.cc:374] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:12.242717 6649 raft_consensus.cc:729] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2f5b88af5f4378ad32ba9cfc7499f8, State: Initialized, Role: FOLLOWER
I20260811 11:50:12.242847 6649 consensus_queue.cc:260] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } }
I20260811 11:50:12.243122 6649 ts_tablet_manager.cc:1428] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260811 11:50:12.295981 6661 raft_consensus.cc:482] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:50:12.296149 6661 raft_consensus.cc:504] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } }
I20260811 11:50:12.297417 6661 leader_election.cc:288] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3640fd03004d415fa5b95cf3a86c9c48 (127.3.221.65:35573), 88db5969cd984867b3e17b6f801fd592 (127.3.221.66:42783)
I20260811 11:50:12.301899 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.297656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:42783 (local address 127.3.221.67:34599)
0811 11:50:12.297894 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:12.297910 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:12.298009 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:12.298400 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:12.298403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:12.298427 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:12.298577 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.298585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.299432 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.299436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.300531 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.300543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.301436 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.301440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.301555 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.301582 (+ 27us) client_negotiation.cc:769] Sending connection context
0811 11:50:12.301668 (+ 86us) client_negotiation.cc:240] Negotiation successful
0811 11:50:12.301748 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":23,"spinlock_wait_cycles":28544}
I20260811 11:50:12.301891 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.298028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:34599 (local address 127.3.221.66:42783)
0811 11:50:12.298148 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:12.298152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:12.298166 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:50:12.298223 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:12.298226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:12.298272 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:12.298362 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:12.298701 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.299320 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.300685 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.301311 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.301438 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.301493 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:12.301691 (+ 198us) server_negotiation.cc:299] Negotiation successful
0811 11:50:12.301764 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 11:50:12.302289 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.297786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:44037 (local address 127.3.221.65:35573)
0811 11:50:12.297913 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:12.297917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:12.298408 (+ 491us) server_negotiation.cc:407] Connection header received
0811 11:50:12.298628 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:12.298631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:12.298682 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:12.298757 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:12.299053 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.299788 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.301099 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.301607 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.301711 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.301760 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:12.302096 (+ 336us) server_negotiation.cc:299] Negotiation successful
0811 11:50:12.302177 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 11:50:12.302476 6321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7e807a584449b484cc23551946861b" candidate_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88db5969cd984867b3e17b6f801fd592" is_pre_election: true
I20260811 11:50:12.302642 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.297656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:35573 (local address 127.3.221.67:44037)
0811 11:50:12.298265 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:12.298276 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:12.298495 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:12.298777 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:12.298780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:12.298800 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:12.298936 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.298942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.299921 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.299925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.300978 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.300987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.301836 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.301851 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.301947 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.301969 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 11:50:12.302433 (+ 464us) client_negotiation.cc:240] Negotiation successful
0811 11:50:12.302507 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":366,"spinlock_wait_cycles":2688,"thread_start_us":85,"threads_started":1}
I20260811 11:50:12.302642 6321 raft_consensus.cc:2454] T 3c7e807a584449b484cc23551946861b P 88db5969cd984867b3e17b6f801fd592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2f5b88af5f4378ad32ba9cfc7499f8 in term 0.
I20260811 11:50:12.303131 6437 leader_election.cc:302] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2f5b88af5f4378ad32ba9cfc7499f8, 88db5969cd984867b3e17b6f801fd592; no voters:
I20260811 11:50:12.303139 6071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7e807a584449b484cc23551946861b" candidate_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3640fd03004d415fa5b95cf3a86c9c48" is_pre_election: true
I20260811 11:50:12.303354 6071 raft_consensus.cc:2454] T 3c7e807a584449b484cc23551946861b P 3640fd03004d415fa5b95cf3a86c9c48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2f5b88af5f4378ad32ba9cfc7499f8 in term 0.
I20260811 11:50:12.303598 6661 raft_consensus.cc:2781] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:50:12.303658 6661 raft_consensus.cc:482] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:50:12.303691 6661 raft_consensus.cc:3037] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:12.304817 6661 raft_consensus.cc:504] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } }
I20260811 11:50:12.305174 6661 leader_election.cc:288] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8 [CANDIDATE]: Term 1 election: Requested vote from peers 3640fd03004d415fa5b95cf3a86c9c48 (127.3.221.65:35573), 88db5969cd984867b3e17b6f801fd592 (127.3.221.66:42783)
I20260811 11:50:12.305526 6071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7e807a584449b484cc23551946861b" candidate_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3640fd03004d415fa5b95cf3a86c9c48"
I20260811 11:50:12.305652 6071 raft_consensus.cc:3037] T 3c7e807a584449b484cc23551946861b P 3640fd03004d415fa5b95cf3a86c9c48 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:12.305671 6321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7e807a584449b484cc23551946861b" candidate_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88db5969cd984867b3e17b6f801fd592"
I20260811 11:50:12.305795 6321 raft_consensus.cc:3037] T 3c7e807a584449b484cc23551946861b P 88db5969cd984867b3e17b6f801fd592 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:12.306849 6071 raft_consensus.cc:2454] T 3c7e807a584449b484cc23551946861b P 3640fd03004d415fa5b95cf3a86c9c48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2f5b88af5f4378ad32ba9cfc7499f8 in term 1.
I20260811 11:50:12.306856 6321 raft_consensus.cc:2454] T 3c7e807a584449b484cc23551946861b P 88db5969cd984867b3e17b6f801fd592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2f5b88af5f4378ad32ba9cfc7499f8 in term 1.
I20260811 11:50:12.307183 6438 leader_election.cc:302] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2f5b88af5f4378ad32ba9cfc7499f8, 3640fd03004d415fa5b95cf3a86c9c48; no voters:
I20260811 11:50:12.307410 6661 raft_consensus.cc:2781] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:12.307610 6661 raft_consensus.cc:686] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 1 LEADER]: Becoming Leader. State: Replica: 1e2f5b88af5f4378ad32ba9cfc7499f8, State: Running, Role: LEADER
I20260811 11:50:12.307860 6661 consensus_queue.cc:237] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } }
I20260811 11:50:12.310289 5822 catalog_manager.cc:5495] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2f5b88af5f4378ad32ba9cfc7499f8 (127.3.221.67). New cstate: current_term: 1 leader_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } health_report { overall_health: HEALTHY } } }
Aug 11 11:50:12 dist-test-slave-bndk krb5kdc[5752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449011, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:12.347913 5898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.339347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55932 (local address 127.3.221.126:41675)
0811 11:50:12.339582 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:12.339587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:12.339844 (+ 257us) server_negotiation.cc:407] Connection header received
0811 11:50:12.339973 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:12.339976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:12.340020 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:12.340088 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:12.340850 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.341423 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.342217 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.342425 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.342513 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.344810 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:12.344813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:12.344815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:12.344838 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:12.346190 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:12.346662 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:12.346666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:12.346667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:12.346749 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:12.347006 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:12.347009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:12.347011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:12.347192 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:12.347280 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:12.347594 (+ 314us) server_negotiation.cc:299] Negotiation successful
0811 11:50:12.347788 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":87}
I20260811 11:50:12.347925 6714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.339244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:41675 (local address 127.0.0.1:55932)
0811 11:50:12.339714 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:12.339733 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:12.339875 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:12.340135 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:12.340138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:12.340599 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:12.340713 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.340721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.341533 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.341536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.342093 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.342102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.342538 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.342541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.342616 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.343187 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:12.343200 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:12.344668 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:12.346304 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:12.346315 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:12.346324 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:12.346563 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:12.346838 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:12.346841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:12.346843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:12.346922 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:12.347304 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:12.347311 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:12.347425 (+ 114us) client_negotiation.cc:769] Sending connection context
0811 11:50:12.347525 (+ 100us) client_negotiation.cc:240] Negotiation successful
0811 11:50:12.347708 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":221,"threads_started":1}
I20260811 11:50:12.352188 5822 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:55932:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 11:50:12.353011 5822 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:50:12.358150 6270 tablet_service.cc:1467] Processing CreateTablet for tablet 1d0cdb293051415db4996c65c2dd969e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fb83a9fa84a84fe0b854ead3599a4d9a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 11:50:12.358402 6518 tablet_service.cc:1467] Processing CreateTablet for tablet 1d0cdb293051415db4996c65c2dd969e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fb83a9fa84a84fe0b854ead3599a4d9a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 11:50:12.358633 6270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d0cdb293051415db4996c65c2dd969e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:12.358795 6518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d0cdb293051415db4996c65c2dd969e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:12.358150 6045 tablet_service.cc:1467] Processing CreateTablet for tablet 1d0cdb293051415db4996c65c2dd969e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fb83a9fa84a84fe0b854ead3599a4d9a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 11:50:12.361413 6045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d0cdb293051415db4996c65c2dd969e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:12.361860 6647 tablet_bootstrap.cc:492] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592: Bootstrap starting.
I20260811 11:50:12.361972 6649 tablet_bootstrap.cc:492] T 1d0cdb293051415db4996c65c2dd969e P 1e2f5b88af5f4378ad32ba9cfc7499f8: Bootstrap starting.
I20260811 11:50:12.362864 6647 tablet_bootstrap.cc:654] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:12.362928 6649 tablet_bootstrap.cc:654] T 1d0cdb293051415db4996c65c2dd969e P 1e2f5b88af5f4378ad32ba9cfc7499f8: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:12.364332 6648 tablet_bootstrap.cc:492] T 1d0cdb293051415db4996c65c2dd969e P 3640fd03004d415fa5b95cf3a86c9c48: Bootstrap starting.
I20260811 11:50:12.365187 6648 tablet_bootstrap.cc:654] T 1d0cdb293051415db4996c65c2dd969e P 3640fd03004d415fa5b95cf3a86c9c48: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:12.365304 6649 tablet_bootstrap.cc:492] T 1d0cdb293051415db4996c65c2dd969e P 1e2f5b88af5f4378ad32ba9cfc7499f8: No bootstrap required, opened a new log
I20260811 11:50:12.365304 6647 tablet_bootstrap.cc:492] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592: No bootstrap required, opened a new log
I20260811 11:50:12.365387 6649 ts_tablet_manager.cc:1397] T 1d0cdb293051415db4996c65c2dd969e P 1e2f5b88af5f4378ad32ba9cfc7499f8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 11:50:12.365392 6647 ts_tablet_manager.cc:1397] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260811 11:50:12.365830 6647 raft_consensus.cc:348] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } }
I20260811 11:50:12.365831 6649 raft_consensus.cc:348] T 1d0cdb293051415db4996c65c2dd969e P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } }
I20260811 11:50:12.365959 6647 raft_consensus.cc:374] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:12.365964 6649 raft_consensus.cc:374] T 1d0cdb293051415db4996c65c2dd969e P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:12.365988 6647 raft_consensus.cc:729] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88db5969cd984867b3e17b6f801fd592, State: Initialized, Role: FOLLOWER
I20260811 11:50:12.365998 6649 raft_consensus.cc:729] T 1d0cdb293051415db4996c65c2dd969e P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2f5b88af5f4378ad32ba9cfc7499f8, State: Initialized, Role: FOLLOWER
I20260811 11:50:12.366106 6647 consensus_queue.cc:260] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } }
I20260811 11:50:12.366120 6649 consensus_queue.cc:260] T 1d0cdb293051415db4996c65c2dd969e P 1e2f5b88af5f4378ad32ba9cfc7499f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } }
I20260811 11:50:12.366382 6647 ts_tablet_manager.cc:1428] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:12.366451 6649 ts_tablet_manager.cc:1428] T 1d0cdb293051415db4996c65c2dd969e P 1e2f5b88af5f4378ad32ba9cfc7499f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:12.366755 6648 tablet_bootstrap.cc:492] T 1d0cdb293051415db4996c65c2dd969e P 3640fd03004d415fa5b95cf3a86c9c48: No bootstrap required, opened a new log
I20260811 11:50:12.366842 6648 ts_tablet_manager.cc:1397] T 1d0cdb293051415db4996c65c2dd969e P 3640fd03004d415fa5b95cf3a86c9c48: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260811 11:50:12.366820 6656 tablet_replica.cc:439] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } } }
I20260811 11:50:12.366969 6656 tablet_replica.cc:442] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:50:12.367184 6661 tablet_replica.cc:439] T 1d0cdb293051415db4996c65c2dd969e P 1e2f5b88af5f4378ad32ba9cfc7499f8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } } }
I20260811 11:50:12.367244 6648 raft_consensus.cc:348] T 1d0cdb293051415db4996c65c2dd969e P 3640fd03004d415fa5b95cf3a86c9c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } }
I20260811 11:50:12.367332 6661 tablet_replica.cc:442] T 1d0cdb293051415db4996c65c2dd969e P 1e2f5b88af5f4378ad32ba9cfc7499f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:50:12.367353 6648 raft_consensus.cc:374] T 1d0cdb293051415db4996c65c2dd969e P 3640fd03004d415fa5b95cf3a86c9c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:12.367388 6648 raft_consensus.cc:729] T 1d0cdb293051415db4996c65c2dd969e P 3640fd03004d415fa5b95cf3a86c9c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3640fd03004d415fa5b95cf3a86c9c48, State: Initialized, Role: FOLLOWER
I20260811 11:50:12.367501 6648 consensus_queue.cc:260] T 1d0cdb293051415db4996c65c2dd969e P 3640fd03004d415fa5b95cf3a86c9c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } }
I20260811 11:50:12.367769 6648 ts_tablet_manager.cc:1428] T 1d0cdb293051415db4996c65c2dd969e P 3640fd03004d415fa5b95cf3a86c9c48: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:12.367978 6660 tablet_replica.cc:439] T 1d0cdb293051415db4996c65c2dd969e P 3640fd03004d415fa5b95cf3a86c9c48: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } } }
I20260811 11:50:12.368093 6660 tablet_replica.cc:442] T 1d0cdb293051415db4996c65c2dd969e P 3640fd03004d415fa5b95cf3a86c9c48: This TxnStatusTablet replica's current role is: FOLLOWER
W20260811 11:50:12.463049 6159 tablet.cc:2369] T 1d0cdb293051415db4996c65c2dd969e P 3640fd03004d415fa5b95cf3a86c9c48: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:50:12.464035 6617 tablet.cc:2369] T 1d0cdb293051415db4996c65c2dd969e P 1e2f5b88af5f4378ad32ba9cfc7499f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:50:12.480890 6395 tablet.cc:2369] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:50:12.717789 6656 raft_consensus.cc:482] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:50:12.717947 6656 raft_consensus.cc:504] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } }
I20260811 11:50:12.719132 6656 leader_election.cc:288] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3640fd03004d415fa5b95cf3a86c9c48 (127.3.221.65:35573), 1e2f5b88af5f4378ad32ba9cfc7499f8 (127.3.221.67:37089)
I20260811 11:50:12.723951 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.719525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:41709 (local address 127.3.221.65:35573)
0811 11:50:12.719661 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:12.719665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:12.719828 (+ 163us) server_negotiation.cc:407] Connection header received
0811 11:50:12.720001 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:12.720005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:12.720068 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:12.720165 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:12.720499 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.721340 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.722787 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.723352 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.723463 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.723520 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:12.723724 (+ 204us) server_negotiation.cc:299] Negotiation successful
0811 11:50:12.723810 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 11:50:12.723994 6720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.719844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:37089 (local address 127.3.221.66:51839)
0811 11:50:12.720124 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:12.720136 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:12.720221 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:12.720470 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:12.720473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:12.720491 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:12.720603 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.720608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.721598 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.721600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.722673 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.722686 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.723578 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.723581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.723659 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.723681 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 11:50:12.723767 (+ 86us) client_negotiation.cc:240] Negotiation successful
0811 11:50:12.723841 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":21,"thread_start_us":58,"threads_started":1}
I20260811 11:50:12.724012 6718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.719417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:35573 (local address 127.3.221.66:41709)
0811 11:50:12.719732 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:12.719747 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:12.719874 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:12.720201 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:12.720204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:12.720230 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:12.720376 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.720382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.721502 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.721506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.722635 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.722649 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.723478 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.723498 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.723593 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.723617 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 11:50:12.723690 (+ 73us) client_negotiation.cc:240] Negotiation successful
0811 11:50:12.723755 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":67,"spinlock_wait_cycles":8960,"thread_start_us":92,"threads_started":1}
I20260811 11:50:12.723999 6722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.719969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:51839 (local address 127.3.221.67:37089)
0811 11:50:12.720221 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:12.720225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:12.720240 (+ 15us) server_negotiation.cc:407] Connection header received
0811 11:50:12.720315 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:12.720318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:12.720365 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:12.720457 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:12.720707 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.721498 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.722820 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.723425 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.723546 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.723601 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:12.723802 (+ 201us) server_negotiation.cc:299] Negotiation successful
0811 11:50:12.723877 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":69,"threads_started":1}
I20260811 11:50:12.724653 6071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0cdb293051415db4996c65c2dd969e" candidate_uuid: "88db5969cd984867b3e17b6f801fd592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3640fd03004d415fa5b95cf3a86c9c48" is_pre_election: true
I20260811 11:50:12.724633 6548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0cdb293051415db4996c65c2dd969e" candidate_uuid: "88db5969cd984867b3e17b6f801fd592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" is_pre_election: true
I20260811 11:50:12.724800 6548 raft_consensus.cc:2454] T 1d0cdb293051415db4996c65c2dd969e P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88db5969cd984867b3e17b6f801fd592 in term 0.
I20260811 11:50:12.724799 6071 raft_consensus.cc:2454] T 1d0cdb293051415db4996c65c2dd969e P 3640fd03004d415fa5b95cf3a86c9c48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88db5969cd984867b3e17b6f801fd592 in term 0.
I20260811 11:50:12.725172 6203 leader_election.cc:302] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3640fd03004d415fa5b95cf3a86c9c48, 88db5969cd984867b3e17b6f801fd592; no voters:
I20260811 11:50:12.725474 6656 raft_consensus.cc:2781] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:50:12.725538 6656 raft_consensus.cc:482] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:50:12.725564 6656 raft_consensus.cc:3037] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:12.726622 6656 raft_consensus.cc:504] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } }
I20260811 11:50:12.726958 6656 leader_election.cc:288] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592 [CANDIDATE]: Term 1 election: Requested vote from peers 3640fd03004d415fa5b95cf3a86c9c48 (127.3.221.65:35573), 1e2f5b88af5f4378ad32ba9cfc7499f8 (127.3.221.67:37089)
I20260811 11:50:12.727349 6071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0cdb293051415db4996c65c2dd969e" candidate_uuid: "88db5969cd984867b3e17b6f801fd592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3640fd03004d415fa5b95cf3a86c9c48"
I20260811 11:50:12.727381 6548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0cdb293051415db4996c65c2dd969e" candidate_uuid: "88db5969cd984867b3e17b6f801fd592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8"
I20260811 11:50:12.727481 6548 raft_consensus.cc:3037] T 1d0cdb293051415db4996c65c2dd969e P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:12.727478 6071 raft_consensus.cc:3037] T 1d0cdb293051415db4996c65c2dd969e P 3640fd03004d415fa5b95cf3a86c9c48 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:12.728507 6071 raft_consensus.cc:2454] T 1d0cdb293051415db4996c65c2dd969e P 3640fd03004d415fa5b95cf3a86c9c48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88db5969cd984867b3e17b6f801fd592 in term 1.
I20260811 11:50:12.728590 6548 raft_consensus.cc:2454] T 1d0cdb293051415db4996c65c2dd969e P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88db5969cd984867b3e17b6f801fd592 in term 1.
I20260811 11:50:12.728812 6203 leader_election.cc:302] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3640fd03004d415fa5b95cf3a86c9c48, 88db5969cd984867b3e17b6f801fd592; no voters:
I20260811 11:50:12.729110 6656 raft_consensus.cc:2781] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:12.729300 6656 raft_consensus.cc:686] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592 [term 1 LEADER]: Becoming Leader. State: Replica: 88db5969cd984867b3e17b6f801fd592, State: Running, Role: LEADER
I20260811 11:50:12.729573 6656 consensus_queue.cc:237] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } }
I20260811 11:50:12.731612 6723 tablet_replica.cc:439] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592: TxnStatusTablet state changed. Reason: New leader 88db5969cd984867b3e17b6f801fd592. Latest consensus state: current_term: 1 leader_uuid: "88db5969cd984867b3e17b6f801fd592" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } } }
I20260811 11:50:12.731732 6723 tablet_replica.cc:442] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592: This TxnStatusTablet replica's current role is: LEADER
I20260811 11:50:12.732108 5821 catalog_manager.cc:5495] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88db5969cd984867b3e17b6f801fd592 (127.3.221.66). New cstate: current_term: 1 leader_uuid: "88db5969cd984867b3e17b6f801fd592" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } health_report { overall_health: UNKNOWN } } }
I20260811 11:50:12.732386 6726 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 11:50:12.732575 6726 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 11:50:12.785389 6661 consensus_queue.cc:1035] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:12.789776 6661 consensus_queue.cc:1035] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:12.821897 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.817607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52962 (local address 127.3.221.66:42783)
0811 11:50:12.817849 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:12.817854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:12.817870 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:50:12.817975 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:12.817978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:12.818038 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:12.818178 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:12.818501 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.819020 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.820256 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.820493 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.820583 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.820908 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:12.821487 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:12.821577 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:12.821701 (+ 124us) server_negotiation.cc:299] Negotiation successful
0811 11:50:12.821763 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":66,"threads_started":1}
I20260811 11:50:12.821919 6714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.817462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:42783 (local address 127.0.0.1:52962)
0811 11:50:12.817701 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:12.817718 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:12.817832 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:12.818221 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:12.818224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:12.818235 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:50:12.818367 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.818374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.819136 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.819140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.820135 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.820149 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.820604 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.820608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.820700 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.820771 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:50:12.821583 (+ 812us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:50:12.821601 (+ 18us) client_negotiation.cc:769] Sending connection context
0811 11:50:12.821675 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 11:50:12.821778 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260811 11:50:12.825346 6548 raft_consensus.cc:1261] T 1d0cdb293051415db4996c65c2dd969e P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 1 FOLLOWER]: Refusing update from remote peer 88db5969cd984867b3e17b6f801fd592: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:12.825795 6071 raft_consensus.cc:1261] T 1d0cdb293051415db4996c65c2dd969e P 3640fd03004d415fa5b95cf3a86c9c48 [term 1 FOLLOWER]: Refusing update from remote peer 88db5969cd984867b3e17b6f801fd592: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:12.825838 6656 consensus_queue.cc:1035] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:12.826251 6656 consensus_queue.cc:1035] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:12.827332 6661 tablet_replica.cc:439] T 1d0cdb293051415db4996c65c2dd969e P 1e2f5b88af5f4378ad32ba9cfc7499f8: TxnStatusTablet state changed. Reason: New leader 88db5969cd984867b3e17b6f801fd592. Latest consensus state: current_term: 1 leader_uuid: "88db5969cd984867b3e17b6f801fd592" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } } }
I20260811 11:50:12.827461 6661 tablet_replica.cc:442] T 1d0cdb293051415db4996c65c2dd969e P 1e2f5b88af5f4378ad32ba9cfc7499f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:50:12.828220 6660 tablet_replica.cc:439] T 1d0cdb293051415db4996c65c2dd969e P 3640fd03004d415fa5b95cf3a86c9c48: TxnStatusTablet state changed. Reason: New leader 88db5969cd984867b3e17b6f801fd592. Latest consensus state: current_term: 1 leader_uuid: "88db5969cd984867b3e17b6f801fd592" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } } }
I20260811 11:50:12.828336 6660 tablet_replica.cc:442] T 1d0cdb293051415db4996c65c2dd969e P 3640fd03004d415fa5b95cf3a86c9c48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:50:12.830040 6723 tablet_replica.cc:439] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88db5969cd984867b3e17b6f801fd592" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } } }
I20260811 11:50:12.830149 6723 tablet_replica.cc:442] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592: This TxnStatusTablet replica's current role is: LEADER
I20260811 11:50:12.830279 6723 tablet_replica.cc:439] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88db5969cd984867b3e17b6f801fd592" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } } }
I20260811 11:50:12.830353 6723 tablet_replica.cc:442] T 1d0cdb293051415db4996c65c2dd969e P 88db5969cd984867b3e17b6f801fd592: This TxnStatusTablet replica's current role is: LEADER
I20260811 11:50:12.830427 6736 mvcc.cc:205] Tried to move back new op lower bound from 7317295156527239168 to 7317295156143226880. Current Snapshot: MvccSnapshot[applied={T|T < 7317295156527239168 or (T in {7317295156527239168})}]
I20260811 11:50:12.831660 6661 tablet_replica.cc:439] T 1d0cdb293051415db4996c65c2dd969e P 1e2f5b88af5f4378ad32ba9cfc7499f8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88db5969cd984867b3e17b6f801fd592" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } } }
I20260811 11:50:12.831784 6661 tablet_replica.cc:442] T 1d0cdb293051415db4996c65c2dd969e P 1e2f5b88af5f4378ad32ba9cfc7499f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:50:12.832052 6660 tablet_replica.cc:439] T 1d0cdb293051415db4996c65c2dd969e P 3640fd03004d415fa5b95cf3a86c9c48: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88db5969cd984867b3e17b6f801fd592" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640fd03004d415fa5b95cf3a86c9c48" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35573 } } peers { permanent_uuid: "88db5969cd984867b3e17b6f801fd592" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 42783 } } peers { permanent_uuid: "1e2f5b88af5f4378ad32ba9cfc7499f8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37089 } } }
I20260811 11:50:12.832161 6660 tablet_replica.cc:442] T 1d0cdb293051415db4996c65c2dd969e P 3640fd03004d415fa5b95cf3a86c9c48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:50:12.840466 6722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.836424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47802 (local address 127.3.221.67:37089)
0811 11:50:12.836544 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:12.836549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:12.836678 (+ 129us) server_negotiation.cc:407] Connection header received
0811 11:50:12.836838 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:12.836840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:12.836895 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:12.836990 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:12.837275 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.837891 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.839070 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.839280 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.839358 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.839637 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:12.840094 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:12.840178 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:12.840283 (+ 105us) server_negotiation.cc:299] Negotiation successful
0811 11:50:12.840337 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 11:50:12.851637 6762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.848032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:42783 (local address 127.3.221.67:50185)
0811 11:50:12.848430 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:12.848442 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:12.848565 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:12.848800 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:12.848803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:12.848811 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:50:12.848909 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.848914 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.849675 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.849679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.850398 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.850411 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.850901 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.850904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.850991 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.851041 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:50:12.851355 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:50:12.851361 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 11:50:12.851422 (+ 61us) client_negotiation.cc:240] Negotiation successful
0811 11:50:12.851512 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":107,"threads_started":1}
I20260811 11:50:12.851632 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.848135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:50185 (local address 127.3.221.66:42783)
0811 11:50:12.848249 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:12.848253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:12.848537 (+ 284us) server_negotiation.cc:407] Connection header received
0811 11:50:12.848664 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:12.848668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:12.848718 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:12.848779 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:12.849019 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.849571 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.850533 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.850787 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.850882 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.851184 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:12.851273 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:12.851333 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:12.851434 (+ 101us) server_negotiation.cc:299] Negotiation successful
0811 11:50:12.851487 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 11:50:12.881413 6722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.875871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:33621 (local address 127.3.221.67:37089)
0811 11:50:12.876018 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:12.876024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:12.877611 (+ 1587us) server_negotiation.cc:407] Connection header received
0811 11:50:12.877711 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:12.877714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:12.877772 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:12.877843 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:12.878269 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.879050 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.880196 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.880520 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.880637 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.880954 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:12.881040 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:12.881106 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:12.881173 (+ 67us) server_negotiation.cc:299] Negotiation successful
0811 11:50:12.881265 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 11:50:12.881446 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.875794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:37089 (local address 127.3.221.66:33621)
0811 11:50:12.877477 (+ 1683us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:12.877493 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:12.877614 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:12.877973 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:12.877976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:12.877986 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:50:12.878133 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.878140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.879201 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.879204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.880071 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:12.880082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.880669 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.880672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:12.880761 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.880817 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:50:12.881113 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:50:12.881117 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:50:12.881161 (+ 44us) client_negotiation.cc:240] Negotiation successful
0811 11:50:12.881287 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1547,"thread_start_us":93,"threads_started":1}
I20260811 11:50:12.910724 5818 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:12.910931 5818 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:12.913486 5818 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c15d68e3f93442eca157324cef4b0441: Sending DeleteTablet for 3 replicas of tablet 3c7e807a584449b484cc23551946861b
I20260811 11:50:12.914215 6518 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7e807a584449b484cc23551946861b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:12 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:47798
I20260811 11:50:12.914240 6270 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7e807a584449b484cc23551946861b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:12 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:52954
I20260811 11:50:12.914392 6045 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7e807a584449b484cc23551946861b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:12 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:41048
I20260811 11:50:12.914657 6772 tablet_replica.cc:330] T 3c7e807a584449b484cc23551946861b P 88db5969cd984867b3e17b6f801fd592: stopping tablet replica
I20260811 11:50:12.914805 6772 raft_consensus.cc:2229] T 3c7e807a584449b484cc23551946861b P 88db5969cd984867b3e17b6f801fd592 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:12.914948 6772 raft_consensus.cc:2258] T 3c7e807a584449b484cc23551946861b P 88db5969cd984867b3e17b6f801fd592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:12.914975 6773 tablet_replica.cc:330] T 3c7e807a584449b484cc23551946861b P 3640fd03004d415fa5b95cf3a86c9c48: stopping tablet replica
I20260811 11:50:12.915194 6773 raft_consensus.cc:2229] T 3c7e807a584449b484cc23551946861b P 3640fd03004d415fa5b95cf3a86c9c48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:12.915373 6773 raft_consensus.cc:2258] T 3c7e807a584449b484cc23551946861b P 3640fd03004d415fa5b95cf3a86c9c48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:12.915683 6771 tablet_replica.cc:330] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8: stopping tablet replica
I20260811 11:50:12.915741 6772 ts_tablet_manager.cc:1905] T 3c7e807a584449b484cc23551946861b P 88db5969cd984867b3e17b6f801fd592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:12.915815 6771 raft_consensus.cc:2229] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:50:12.916306 6773 ts_tablet_manager.cc:1905] T 3c7e807a584449b484cc23551946861b P 3640fd03004d415fa5b95cf3a86c9c48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:12.916425 6771 raft_consensus.cc:2258] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:12.917147 6771 ts_tablet_manager.cc:1905] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:12.918982 6772 ts_tablet_manager.cc:1918] T 3c7e807a584449b484cc23551946861b P 88db5969cd984867b3e17b6f801fd592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:50:12.919072 6772 log.cc:1196] T 3c7e807a584449b484cc23551946861b P 88db5969cd984867b3e17b6f801fd592: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/3c7e807a584449b484cc23551946861b
I20260811 11:50:12.919339 6772 ts_tablet_manager.cc:1939] T 3c7e807a584449b484cc23551946861b P 88db5969cd984867b3e17b6f801fd592: Deleting consensus metadata
I20260811 11:50:12.920147 5800 catalog_manager.cc:4841] TS 88db5969cd984867b3e17b6f801fd592 (127.3.221.66:42783): tablet 3c7e807a584449b484cc23551946861b (table test-table [id=8488453205514b238f26a4472d47029d]) successfully deleted
I20260811 11:50:12.920267 6771 ts_tablet_manager.cc:1918] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:50:12.920353 6771 log.cc:1196] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/3c7e807a584449b484cc23551946861b
I20260811 11:50:12.920645 6773 ts_tablet_manager.cc:1918] T 3c7e807a584449b484cc23551946861b P 3640fd03004d415fa5b95cf3a86c9c48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:50:12.920647 6771 ts_tablet_manager.cc:1939] T 3c7e807a584449b484cc23551946861b P 1e2f5b88af5f4378ad32ba9cfc7499f8: Deleting consensus metadata
I20260811 11:50:12.920781 6773 log.cc:1196] T 3c7e807a584449b484cc23551946861b P 3640fd03004d415fa5b95cf3a86c9c48: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786449004114841-3957-0/minicluster-data/ts-0/wal/wals/3c7e807a584449b484cc23551946861b
I20260811 11:50:12.921092 6773 ts_tablet_manager.cc:1939] T 3c7e807a584449b484cc23551946861b P 3640fd03004d415fa5b95cf3a86c9c48: Deleting consensus metadata
I20260811 11:50:12.921518 5802 catalog_manager.cc:4841] TS 1e2f5b88af5f4378ad32ba9cfc7499f8 (127.3.221.67:37089): tablet 3c7e807a584449b484cc23551946861b (table test-table [id=8488453205514b238f26a4472d47029d]) successfully deleted
I20260811 11:50:12.921954 5802 catalog_manager.cc:4841] TS 3640fd03004d415fa5b95cf3a86c9c48 (127.3.221.65:35573): tablet 3c7e807a584449b484cc23551946861b (table test-table [id=8488453205514b238f26a4472d47029d]) successfully deleted
Aug 11 11:50:12 dist-test-slave-bndk krb5kdc[5752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449011, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.221.65@KRBTEST.COM
I20260811 11:50:12.931864 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.920805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41064 (local address 127.3.221.65:35573)
0811 11:50:12.920922 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:12.920927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:12.921648 (+ 721us) server_negotiation.cc:407] Connection header received
0811 11:50:12.921830 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:12.921833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:12.921886 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:12.921960 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:12.922761 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.923408 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.924395 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.924635 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.924747 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.928287 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:12.928301 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:12.928319 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:12.928349 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:12.930013 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:12.930538 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:12.930544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:12.930545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:12.930628 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:12.930886 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:12.930888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:12.930889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:12.931159 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:12.931334 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:12.931568 (+ 234us) server_negotiation.cc:299] Negotiation successful
0811 11:50:12.931690 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260811 11:50:12.932492 6144 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41064
I20260811 11:50:12.942803 6783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:12.934394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55946 (local address 127.3.221.126:41675)
0811 11:50:12.934697 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:12.934702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:12.934718 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:50:12.934765 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:12.934768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:12.934818 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:12.934906 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:12.935599 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.936471 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.937376 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:12.937734 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:12.937836 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:12.939289 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:12.939293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:12.939296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:12.939323 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:12.940936 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:12.941477 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:12.941481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:12.941483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:12.941585 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:12.941920 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:12.941923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:12.941925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:12.942155 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:12.942334 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:12.942475 (+ 141us) server_negotiation.cc:299] Negotiation successful
0811 11:50:12.942614 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":107,"threads_started":1}
W20260811 11:50:12.943305 5818 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55946
I20260811 11:50:12.944844 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 5945
I20260811 11:50:12.961812 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 6173
I20260811 11:50:12.977952 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 6415
I20260811 11:50:12.987003 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 5772
2026-08-11T11:50:13Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3330 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:50:13 dist-test-slave-bndk krb5kdc[6788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:50:13 dist-test-slave-bndk krb5kdc[6788](info): set up 2 sockets
Aug 11 11:50:13 dist-test-slave-bndk krb5kdc[6788](info): commencing operation
krb5kdc: starting...
W20260811 11:50:14.165138 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.005s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:50:14 dist-test-slave-bndk krb5kdc[6788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:50:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:50:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:50:14.301072 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:36733
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:39335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:36733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:14.419414 6808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:14.419709 6808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:14.419768 6808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:14.423893 6808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:50:14.423964 6808 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 11:50:14.423986 6808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:14.424010 6808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:14.424032 6808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:50:14.424062 6808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:50:14.429548 6808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:39335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:36733
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:36733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:14.430671 6808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:14.431761 6808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:14.440366 6822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:14.440459 6823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:14.440385 6827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:14.440639 6808 server_base.cc:1034] running on GCE node
I20260811 11:50:14.441437 6808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:14.442586 6808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:14.443805 6808 hybrid_clock.cc:648] HybridClock initialized: now 1786449014443788 us; error 30 us; skew 500 ppm
Aug 11 11:50:14 dist-test-slave-bndk krb5kdc[6788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449014, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:14.451294 6808 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:50:14.452332 6808 webserver.cc:460] Webserver started at http://127.3.221.126:34893/ using document root <none> and password file <none>
I20260811 11:50:14.452924 6808 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:14.452998 6808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:14.453207 6808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:14.455395 6808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "138578b03cf54dfca455d1456d96ac0a"
format_stamp: "Formatted at 2026-08-11 11:50:14 on dist-test-slave-bndk"
server_key: "9d9c68a3760e9accaa1eda60633b888f"
server_key_iv: "02d66a811b12056f566a6eb8c8d0464e"
server_key_version: "encryptionkey@0"
I20260811 11:50:14.455977 6808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "138578b03cf54dfca455d1456d96ac0a"
format_stamp: "Formatted at 2026-08-11 11:50:14 on dist-test-slave-bndk"
server_key: "9d9c68a3760e9accaa1eda60633b888f"
server_key_iv: "02d66a811b12056f566a6eb8c8d0464e"
server_key_version: "encryptionkey@0"
I20260811 11:50:14.459976 6808 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260811 11:50:14.462606 6839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:14.463900 6808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 11:50:14.464028 6808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "138578b03cf54dfca455d1456d96ac0a"
format_stamp: "Formatted at 2026-08-11 11:50:14 on dist-test-slave-bndk"
server_key: "9d9c68a3760e9accaa1eda60633b888f"
server_key_iv: "02d66a811b12056f566a6eb8c8d0464e"
server_key_version: "encryptionkey@0"
I20260811 11:50:14.464141 6808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:14.482180 6808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:14.485412 6808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:14.485617 6808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:14.493965 6808 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:36733
I20260811 11:50:14.494027 6943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:36733 every 8 connection(s)
I20260811 11:50:14.495029 6808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:50:14.498066 6944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:14.499439 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 6808
I20260811 11:50:14.499558 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:50:14.499867 3957 external_mini_cluster.cc:1428] Setting key b7b642895c24b0e68034f04a4911a2a5
I20260811 11:50:14.504714 6944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a: Bootstrap starting.
I20260811 11:50:14.507100 6944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:14.507952 6944 log.cc:824] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a: Log is configured to *not* fsync() on all Append() calls
Aug 11 11:50:14 dist-test-slave-bndk krb5kdc[6788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:14.510268 6944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a: No bootstrap required, opened a new log
I20260811 11:50:14.514051 6944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138578b03cf54dfca455d1456d96ac0a" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36733 } }
I20260811 11:50:14.514297 6944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:14.514336 6944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138578b03cf54dfca455d1456d96ac0a, State: Initialized, Role: FOLLOWER
I20260811 11:50:14.514427 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:14.501522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55808 (local address 127.3.221.126:36733)
0811 11:50:14.502063 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:14.502081 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:14.502121 (+ 40us) server_negotiation.cc:407] Connection header received
0811 11:50:14.502838 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:14.502859 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:14.503247 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:14.503580 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:14.504432 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:14.505517 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:14.506376 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:14.506876 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:14.507038 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:14.509741 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:14.509747 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:14.509762 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:14.509790 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:14.511487 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:14.511936 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:14.511941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:14.511946 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:14.512064 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:14.512343 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:14.512346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:14.512347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:14.512762 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:14.512969 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:14.513292 (+ 323us) server_negotiation.cc:299] Negotiation successful
0811 11:50:14.513602 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":161,"threads_started":1}
I20260811 11:50:14.514837 6944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138578b03cf54dfca455d1456d96ac0a" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36733 } }
I20260811 11:50:14.514976 6944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:50:14.515040 6944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:50:14.515137 6944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:14.516067 6944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138578b03cf54dfca455d1456d96ac0a" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36733 } }
I20260811 11:50:14.516417 6944 leader_election.cc:302] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 138578b03cf54dfca455d1456d96ac0a; no voters:
I20260811 11:50:14.516779 6944 leader_election.cc:288] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:50:14.516875 6952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:14.517071 6952 raft_consensus.cc:686] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a [term 1 LEADER]: Becoming Leader. State: Replica: 138578b03cf54dfca455d1456d96ac0a, State: Running, Role: LEADER
I20260811 11:50:14.517408 6952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138578b03cf54dfca455d1456d96ac0a" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36733 } }
I20260811 11:50:14.518054 6944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:50:14.519057 6954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 138578b03cf54dfca455d1456d96ac0a. Latest consensus state: current_term: 1 leader_uuid: "138578b03cf54dfca455d1456d96ac0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138578b03cf54dfca455d1456d96ac0a" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36733 } } }
I20260811 11:50:14.519196 6954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:14.519572 6964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:50:14.519556 6953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "138578b03cf54dfca455d1456d96ac0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138578b03cf54dfca455d1456d96ac0a" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36733 } } }
I20260811 11:50:14.519766 6953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:14.522754 6964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:50:14.528927 6964 catalog_manager.cc:1347] Generated new cluster ID: a96b22f030ab4b6a9137a1cca3070c48
I20260811 11:50:14.529012 6964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:50:14.543176 6964 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:50:14.544541 6964 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:50:14.552265 6964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a: Generated new TSK 0
I20260811 11:50:14.553155 6964 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
I20260811 11:50:14.588855 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36733
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:39335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:14.705947 6983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:14.706243 6983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:14.706313 6983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:14.710440 6983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:14.710518 6983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:14.710577 6983 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 11:50:14.710605 6983 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 11:50:14.710657 6983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:50:14.716185 6983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:39335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.221.126:36733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:14.717381 6983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:14.718505 6983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:14.728700 6992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:14.728698 6993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:14.728693 6999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:14.728991 6983 server_base.cc:1034] running on GCE node
I20260811 11:50:14.729490 6983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:14.730182 6983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:14.731349 6983 hybrid_clock.cc:648] HybridClock initialized: now 1786449014731331 us; error 32 us; skew 500 ppm
Aug 11 11:50:14 dist-test-slave-bndk krb5kdc[6788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449014, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:14.734279 6983 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:50:14.735239 6983 webserver.cc:460] Webserver started at http://127.3.221.65:43905/ using document root <none> and password file <none>
I20260811 11:50:14.735817 6983 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:14.735889 6983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:14.736112 6983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:14.738281 6983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "de1c7edbf94e40bb8e509194b138cf56"
format_stamp: "Formatted at 2026-08-11 11:50:14 on dist-test-slave-bndk"
server_key: "e1e55bdde8bfab6243cd16ad1a76ae2c"
server_key_iv: "db0693204e07dbf9dbf41b369aa42729"
server_key_version: "encryptionkey@0"
I20260811 11:50:14.738855 6983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "de1c7edbf94e40bb8e509194b138cf56"
format_stamp: "Formatted at 2026-08-11 11:50:14 on dist-test-slave-bndk"
server_key: "e1e55bdde8bfab6243cd16ad1a76ae2c"
server_key_iv: "db0693204e07dbf9dbf41b369aa42729"
server_key_version: "encryptionkey@0"
I20260811 11:50:14.742923 6983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 11:50:14.745559 7008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:14.746923 6983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 11:50:14.747056 6983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "de1c7edbf94e40bb8e509194b138cf56"
format_stamp: "Formatted at 2026-08-11 11:50:14 on dist-test-slave-bndk"
server_key: "e1e55bdde8bfab6243cd16ad1a76ae2c"
server_key_iv: "db0693204e07dbf9dbf41b369aa42729"
server_key_version: "encryptionkey@0"
I20260811 11:50:14.747164 6983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:14.767091 6983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:14.770259 6983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:14.770455 6983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:14.772054 6983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:14.772109 6983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:14.772166 6983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:14.772193 6983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 11:50:14 dist-test-slave-bndk krb5kdc[6788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449014, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:14.783133 6983 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:41349
I20260811 11:50:14.783171 7218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:41349 every 8 connection(s)
I20260811 11:50:14.784078 6983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:50:14.785879 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:14.772852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:60267 (local address 127.3.221.126:36733)
0811 11:50:14.772974 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:14.772978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:14.774131 (+ 1153us) server_negotiation.cc:407] Connection header received
0811 11:50:14.775179 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:14.775183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:14.775258 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:14.775321 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:14.776794 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:14.777387 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:14.778657 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:14.778899 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:14.779006 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:14.782181 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:14.782184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:14.782186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:14.782214 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:14.783670 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:14.784269 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:14.784273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:14.784274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:14.784354 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:14.784742 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:14.784745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:14.784747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:14.784989 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:14.785092 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:14.785631 (+ 539us) server_negotiation.cc:299] Negotiation successful
0811 11:50:14.785752 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 11:50:14.786669 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:14.773468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36733 (local address 127.3.221.65:60267)
0811 11:50:14.773969 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:14.774016 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:14.774918 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:14.775730 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:14.775738 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:14.776084 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:14.776633 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:14.776649 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:14.777514 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:14.777516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:14.778532 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:14.778543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:14.779084 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:14.779091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:14.779336 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:14.780070 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:14.780092 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:14.781986 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:14.783802 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:14.783808 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:14.783823 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:14.784125 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:14.784447 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:14.784450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:14.784452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:14.784627 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:14.785093 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:14.785098 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:14.785382 (+ 284us) client_negotiation.cc:769] Sending connection context
0811 11:50:14.785570 (+ 188us) client_negotiation.cc:240] Negotiation successful
0811 11:50:14.785843 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":142,"threads_started":1}
I20260811 11:50:14.787082 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 6983
I20260811 11:50:14.787209 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:50:14.787544 3957 external_mini_cluster.cc:1428] Setting key cbcf71f7c295814869e73c87305c8406
I20260811 11:50:14.788185 7221 heartbeater.cc:347] Connected to a master server at 127.3.221.126:36733
I20260811 11:50:14.788342 7221 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:14.788926 7221 heartbeater.cc:511] Master 127.3.221.126:36733 requested a full tablet report, sending...
I20260811 11:50:14.790460 6866 ts_manager.cc:194] Registered new tserver with Master: de1c7edbf94e40bb8e509194b138cf56 (127.3.221.65:41349)
I20260811 11:50:14.791999 6866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:60267
I20260811 11:50:14.797945 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:14.790545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36733 (local address 127.3.221.65:59185)
0811 11:50:14.790886 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:14.790900 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:14.791013 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:14.791262 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:14.791265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:14.791491 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:14.791625 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:14.791632 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:14.792376 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:14.792379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:14.793117 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:14.793127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:14.793652 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:14.793656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:14.793748 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:14.794267 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:14.794281 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:14.794677 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:14.796463 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:14.796467 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:14.796468 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:14.796673 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:14.796968 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:14.796970 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:14.796971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:14.797042 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:14.797502 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:14.797504 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:14.797589 (+ 85us) client_negotiation.cc:769] Sending connection context
0811 11:50:14.797670 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 11:50:14.797785 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":97,"threads_started":1}
I20260811 11:50:14.798075 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:14.790621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:59185 (local address 127.3.221.126:36733)
0811 11:50:14.790728 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:14.790733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:14.790997 (+ 264us) server_negotiation.cc:407] Connection header received
0811 11:50:14.791089 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:14.791091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:14.791135 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:14.791228 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:14.791749 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:14.792249 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:14.793248 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:14.793521 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:14.793618 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:14.794831 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:14.794835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:14.794838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:14.794864 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:14.796350 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:14.796779 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:14.796783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:14.796785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:14.796881 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:14.797121 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:14.797125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:14.797127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:14.797412 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:14.797518 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:14.797778 (+ 260us) server_negotiation.cc:299] Negotiation successful
0811 11:50:14.797907 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.3.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
I20260811 11:50:14.820091 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36733
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:39335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:14.935236 7237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:14.935554 7237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:14.935613 7237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:14.939661 7237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:14.939726 7237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:14.939771 7237 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 11:50:14.939791 7237 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 11:50:14.939823 7237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:50:14.945513 7237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:39335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.221.126:36733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:14.946625 7237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:14.947643 7237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:14.957559 7247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:14.957618 7249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:14.957728 7237 server_base.cc:1034] running on GCE node
W20260811 11:50:14.957554 7245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:14.958304 7237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:14.958945 7237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:14.960144 7237 hybrid_clock.cc:648] HybridClock initialized: now 1786449014960120 us; error 30 us; skew 500 ppm
Aug 11 11:50:14 dist-test-slave-bndk krb5kdc[6788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449014, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:14.963020 7237 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:50:14.963971 7237 webserver.cc:460] Webserver started at http://127.3.221.66:44355/ using document root <none> and password file <none>
I20260811 11:50:14.964533 7237 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:14.964597 7237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:14.964777 7237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:14.966979 7237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "eaf3929d5afa44cdb1aac4e306a3d835"
format_stamp: "Formatted at 2026-08-11 11:50:14 on dist-test-slave-bndk"
server_key: "126bfe83b12cdc4848e521d10d4f34b2"
server_key_iv: "25bb8c97506d8bd6a97ad5d6aeecc97b"
server_key_version: "encryptionkey@0"
I20260811 11:50:14.967533 7237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "eaf3929d5afa44cdb1aac4e306a3d835"
format_stamp: "Formatted at 2026-08-11 11:50:14 on dist-test-slave-bndk"
server_key: "126bfe83b12cdc4848e521d10d4f34b2"
server_key_iv: "25bb8c97506d8bd6a97ad5d6aeecc97b"
server_key_version: "encryptionkey@0"
I20260811 11:50:14.972102 7237 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 11:50:14.974767 7264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:14.976120 7237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 11:50:14.976244 7237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "eaf3929d5afa44cdb1aac4e306a3d835"
format_stamp: "Formatted at 2026-08-11 11:50:14 on dist-test-slave-bndk"
server_key: "126bfe83b12cdc4848e521d10d4f34b2"
server_key_iv: "25bb8c97506d8bd6a97ad5d6aeecc97b"
server_key_version: "encryptionkey@0"
I20260811 11:50:14.976379 7237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:15.007308 7237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:15.010442 7237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:15.010634 7237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:15.012239 7237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:15.012290 7237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:15.012346 7237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:15.012382 7237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 11:50:15 dist-test-slave-bndk krb5kdc[6788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449014, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:15.023192 7237 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:45007
I20260811 11:50:15.024283 7237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:50:15.025046 7454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:45007 every 8 connection(s)
I20260811 11:50:15.026923 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.013084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:59987 (local address 127.3.221.126:36733)
0811 11:50:15.013205 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:15.013209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:15.014366 (+ 1157us) server_negotiation.cc:407] Connection header received
0811 11:50:15.015428 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:15.015432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:15.015511 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:15.015572 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:15.017385 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.018207 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.019761 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.020183 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.020295 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.023207 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:15.023212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:15.023214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:15.023249 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:15.024666 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:15.025319 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:15.025323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:15.025325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:15.025419 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:15.025767 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:15.025771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:15.025773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:15.025983 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:15.026088 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:15.026612 (+ 524us) server_negotiation.cc:299] Negotiation successful
0811 11:50:15.026757 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 11:50:15.027717 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.013686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36733 (local address 127.3.221.66:59987)
0811 11:50:15.014207 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:15.014252 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:15.015150 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:15.016029 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:15.016039 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:15.016451 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:15.017176 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.017198 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.018353 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.018357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.019632 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.019647 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.020353 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.020356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.020563 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.021235 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:15.021258 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:15.022984 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:15.024805 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:15.024811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:15.024826 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:15.025175 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:15.025515 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:15.025518 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:15.025521 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:15.025658 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:15.026095 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:15.026101 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:15.026351 (+ 250us) client_negotiation.cc:769] Sending connection context
0811 11:50:15.026524 (+ 173us) client_negotiation.cc:240] Negotiation successful
0811 11:50:15.026856 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":164,"threads_started":1}
I20260811 11:50:15.027936 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 7237
I20260811 11:50:15.028061 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:50:15.028388 3957 external_mini_cluster.cc:1428] Setting key 3841d4a99b06f66262cf0bfb27651e98
I20260811 11:50:15.029007 7455 heartbeater.cc:347] Connected to a master server at 127.3.221.126:36733
I20260811 11:50:15.029155 7455 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:15.029775 7455 heartbeater.cc:511] Master 127.3.221.126:36733 requested a full tablet report, sending...
I20260811 11:50:15.032593 6863 ts_manager.cc:194] Registered new tserver with Master: eaf3929d5afa44cdb1aac4e306a3d835 (127.3.221.66:45007)
I20260811 11:50:15.033401 6863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:59987
I20260811 11:50:15.039403 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.032007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:33299 (local address 127.3.221.126:36733)
0811 11:50:15.032117 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:15.032121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:15.032133 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:50:15.032162 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:15.032164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:15.032197 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:15.032262 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:15.032790 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.033580 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.034689 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.035034 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.035137 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.036324 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:15.036328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:15.036331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:15.036357 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:15.037782 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:15.038210 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:15.038214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:15.038215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:15.038296 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:15.038565 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:15.038569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:15.038570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:15.038792 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:15.038879 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:15.039140 (+ 261us) server_negotiation.cc:299] Negotiation successful
0811 11:50:15.039246 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 11:50:15.039428 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.031520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36733 (local address 127.3.221.66:33299)
0811 11:50:15.031876 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:15.031892 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:15.032000 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:15.032301 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:15.032303 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:15.032529 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:15.032653 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.032661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.033744 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.033747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.034544 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.034557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.035177 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.035181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.035277 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.035809 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:15.035824 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:15.036183 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:15.037899 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:15.037903 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:15.037905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:15.038114 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:15.038384 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:15.038388 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:15.038390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:15.038479 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:15.038892 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:15.038894 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:15.038985 (+ 91us) client_negotiation.cc:769] Sending connection context
0811 11:50:15.039073 (+ 88us) client_negotiation.cc:240] Negotiation successful
0811 11:50:15.039225 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":167,"threads_started":1}
WARNING: no policy specified for kudu/127.3.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
I20260811 11:50:15.060734 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36733
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:39335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:15.179546 7469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:15.179836 7469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:15.179885 7469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:15.184000 7469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:15.184065 7469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:15.184106 7469 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 11:50:15.184125 7469 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 11:50:15.184157 7469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:50:15.189652 7469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:39335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.221.126:36733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:15.190735 7469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:15.191768 7469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:15.201843 7482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:15.201913 7469 server_base.cc:1034] running on GCE node
W20260811 11:50:15.201843 7490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:15.201862 7483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:15.202629 7469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:15.203316 7469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:15.204480 7469 hybrid_clock.cc:648] HybridClock initialized: now 1786449015204468 us; error 37 us; skew 500 ppm
Aug 11 11:50:15 dist-test-slave-bndk krb5kdc[6788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449015, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:15.207381 7469 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:50:15.208415 7469 webserver.cc:460] Webserver started at http://127.3.221.67:44691/ using document root <none> and password file <none>
I20260811 11:50:15.208989 7469 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:15.209059 7469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:15.209311 7469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:15.211404 7469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "c4edf677c3f24f6a9b0d7054252810ad"
format_stamp: "Formatted at 2026-08-11 11:50:15 on dist-test-slave-bndk"
server_key: "027525a1857e4f4c155a2468d9d01660"
server_key_iv: "490a49f2893d1cd97023f80b600fc1d5"
server_key_version: "encryptionkey@0"
I20260811 11:50:15.211980 7469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "c4edf677c3f24f6a9b0d7054252810ad"
format_stamp: "Formatted at 2026-08-11 11:50:15 on dist-test-slave-bndk"
server_key: "027525a1857e4f4c155a2468d9d01660"
server_key_iv: "490a49f2893d1cd97023f80b600fc1d5"
server_key_version: "encryptionkey@0"
I20260811 11:50:15.215811 7469 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 11:50:15.218556 7501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:15.219978 7469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 11:50:15.220098 7469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "c4edf677c3f24f6a9b0d7054252810ad"
format_stamp: "Formatted at 2026-08-11 11:50:15 on dist-test-slave-bndk"
server_key: "027525a1857e4f4c155a2468d9d01660"
server_key_iv: "490a49f2893d1cd97023f80b600fc1d5"
server_key_version: "encryptionkey@0"
I20260811 11:50:15.220204 7469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:15.237026 7469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:15.240389 7469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:15.240582 7469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:15.242249 7469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:15.242305 7469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:15.242360 7469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:15.242391 7469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 11:50:15 dist-test-slave-bndk krb5kdc[6788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449015, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:15.252580 7469 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:43685
I20260811 11:50:15.252660 7688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:43685 every 8 connection(s)
I20260811 11:50:15.253755 7469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:50:15.256588 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.243006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:48551 (local address 127.3.221.126:36733)
0811 11:50:15.243197 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:15.243203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:15.244120 (+ 917us) server_negotiation.cc:407] Connection header received
0811 11:50:15.245233 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:15.245238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:15.245303 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:15.245422 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:15.246976 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.247799 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.249542 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.249778 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.249877 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.252685 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:15.252689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:15.252692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:15.252723 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:15.254403 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:15.255008 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:15.255012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:15.255014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:15.255105 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:15.255448 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:15.255451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:15.255453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:15.255653 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:15.255742 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:15.256273 (+ 531us) server_negotiation.cc:299] Negotiation successful
0811 11:50:15.256421 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 11:50:15.257440 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.243435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36733 (local address 127.3.221.67:48551)
0811 11:50:15.243926 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:15.243982 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:15.244965 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:15.245617 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:15.245627 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:15.246069 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:15.246798 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.246821 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.247937 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.247941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.249410 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.249423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.249942 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.249945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.250129 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.250754 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:15.250773 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:15.252493 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:15.254529 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:15.254535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:15.254551 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:15.254901 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:15.255199 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:15.255202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:15.255204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:15.255346 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:15.255762 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:15.255769 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:15.256031 (+ 262us) client_negotiation.cc:769] Sending connection context
0811 11:50:15.256203 (+ 172us) client_negotiation.cc:240] Negotiation successful
0811 11:50:15.256541 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":111,"threads_started":1}
I20260811 11:50:15.258600 7693 heartbeater.cc:347] Connected to a master server at 127.3.221.126:36733
I20260811 11:50:15.258760 7693 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:15.259033 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 7469
I20260811 11:50:15.259143 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:50:15.259374 7693 heartbeater.cc:511] Master 127.3.221.126:36733 requested a full tablet report, sending...
I20260811 11:50:15.259457 3957 external_mini_cluster.cc:1428] Setting key 285f0f8baf5465663f700e42f3fa3c4a
I20260811 11:50:15.260598 6866 ts_manager.cc:194] Registered new tserver with Master: c4edf677c3f24f6a9b0d7054252810ad (127.3.221.67:43685)
I20260811 11:50:15.261337 6866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:48551
I20260811 11:50:15.261616 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:50:15.268961 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.261068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36733 (local address 127.3.221.67:43203)
0811 11:50:15.261624 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:15.261638 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:15.261821 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:15.262101 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:15.262105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:15.262336 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:15.262463 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.262472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.263435 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.263438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.264052 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.264063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.264508 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.264512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.264602 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.265147 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:15.265161 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:15.265637 (+ 476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:15.267324 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:15.267328 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:15.267330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:15.267621 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:15.267965 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:15.267967 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:15.267969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:15.268054 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:15.268512 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:15.268514 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:15.268602 (+ 88us) client_negotiation.cc:769] Sending connection context
0811 11:50:15.268681 (+ 79us) client_negotiation.cc:240] Negotiation successful
0811 11:50:15.268800 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":104,"threads_started":1}
I20260811 11:50:15.269086 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.261140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:43203 (local address 127.3.221.126:36733)
0811 11:50:15.261286 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:15.261290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:15.261727 (+ 437us) server_negotiation.cc:407] Connection header received
0811 11:50:15.261939 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:15.261942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:15.261990 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:15.262070 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:15.262597 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.263282 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.264177 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.264397 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.264476 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.265782 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:15.265786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:15.265788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:15.265812 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:15.267196 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:15.267721 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:15.267726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:15.267728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:15.267855 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:15.268157 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:15.268161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:15.268163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:15.268419 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:15.268528 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:15.268742 (+ 214us) server_negotiation.cc:299] Negotiation successful
0811 11:50:15.268879 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Aug 11 11:50:15 dist-test-slave-bndk krb5kdc[6788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 11:50:15 dist-test-slave-bndk krb5kdc[6788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:15.290949 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.281893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55814 (local address 127.3.221.126:36733)
0811 11:50:15.282013 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:15.282016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:15.282030 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:50:15.282075 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:15.282077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:15.282126 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:15.282210 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:15.282948 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.283508 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.284442 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.284670 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.284767 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.287341 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:15.287346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:15.287348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:15.287382 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:15.288873 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:15.289398 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:15.289401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:15.289402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:15.289501 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:15.289875 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:15.289879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:15.289880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:15.290118 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:15.290245 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:15.290671 (+ 426us) server_negotiation.cc:299] Negotiation successful
0811 11:50:15.290801 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 11:50:15.294565 6866 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:50:15.297821 6866 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:50:15.313197 7719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.307373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39378 (local address 127.3.221.65:41349)
0811 11:50:15.308450 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:15.308458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:15.308483 (+ 25us) server_negotiation.cc:407] Connection header received
0811 11:50:15.308565 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:15.308571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:15.308799 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:15.309358 (+ 559us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:15.309503 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.310258 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.311816 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.312512 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.312613 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.312778 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:15.312955 (+ 177us) server_negotiation.cc:299] Negotiation successful
0811 11:50:15.313056 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":938,"thread_start_us":72,"threads_started":1}
I20260811 11:50:15.313197 7715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.307246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:41349 (local address 127.0.0.1:39378)
0811 11:50:15.307832 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:15.307888 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:15.308036 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:15.308981 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:15.308987 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:15.309009 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:15.309173 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.309180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.310384 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.310389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.311670 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.311683 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.312636 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.312640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.312751 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.312790 (+ 39us) client_negotiation.cc:769] Sending connection context
0811 11:50:15.312900 (+ 110us) client_negotiation.cc:240] Negotiation successful
0811 11:50:15.313037 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":3,"thread_start_us":89,"threads_started":1}
I20260811 11:50:15.313767 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.307545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:45007 (local address 127.0.0.1:54104)
0811 11:50:15.308529 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:15.308540 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:15.308656 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:15.309187 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:15.309190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:15.309208 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:15.309356 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.309364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.310839 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.310843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.312057 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.312068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.313324 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.313328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.313441 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.313465 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 11:50:15.313543 (+ 78us) client_negotiation.cc:240] Negotiation successful
0811 11:50:15.313614 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"mutex_wait_us":101,"thread_start_us":107,"threads_started":1}
I20260811 11:50:15.313846 7723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.308093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54104 (local address 127.3.221.66:45007)
0811 11:50:15.308555 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:15.308568 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:15.308747 (+ 179us) server_negotiation.cc:407] Connection header received
0811 11:50:15.308823 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:15.308829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:15.309012 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:15.309183 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:15.309505 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.310696 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.312166 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.312967 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.313068 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.313256 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:15.313609 (+ 353us) server_negotiation.cc:299] Negotiation successful
0811 11:50:15.313711 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":114,"threads_started":1}
I20260811 11:50:15.314193 7722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.308083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49786 (local address 127.3.221.67:43685)
0811 11:50:15.308758 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:15.308769 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:15.308793 (+ 24us) server_negotiation.cc:407] Connection header received
0811 11:50:15.308874 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:15.308878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:15.309071 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:15.309244 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:15.309607 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.310722 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.312122 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.312955 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.313068 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.313253 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:15.313970 (+ 717us) server_negotiation.cc:299] Negotiation successful
0811 11:50:15.314080 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":128,"threads_started":1}
I20260811 11:50:15.313778 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.307335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:43685 (local address 127.0.0.1:49786)
0811 11:50:15.308257 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:15.308271 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:15.308356 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:15.309379 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:15.309382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:15.309399 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:15.309479 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.309485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.310849 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.310852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.311994 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.312004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.313453 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.313468 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.313544 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.313564 (+ 20us) client_negotiation.cc:769] Sending connection context
0811 11:50:15.313621 (+ 57us) client_negotiation.cc:240] Negotiation successful
0811 11:50:15.313676 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"mutex_wait_us":4,"thread_start_us":129,"threads_started":1}
I20260811 11:50:15.316605 7098 tablet_service.cc:1467] Processing CreateTablet for tablet b4c306d3a03b4d5385e2093349e66a45 (DEFAULT_TABLE table=test-table [id=0a82454f45914818a503624a2cbe0459]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:15.316792 7354 tablet_service.cc:1467] Processing CreateTablet for tablet b4c306d3a03b4d5385e2093349e66a45 (DEFAULT_TABLE table=test-table [id=0a82454f45914818a503624a2cbe0459]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:15.317130 7580 tablet_service.cc:1467] Processing CreateTablet for tablet b4c306d3a03b4d5385e2093349e66a45 (DEFAULT_TABLE table=test-table [id=0a82454f45914818a503624a2cbe0459]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:15.317937 7098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4c306d3a03b4d5385e2093349e66a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:15.318035 7354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4c306d3a03b4d5385e2093349e66a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:15.318317 7580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4c306d3a03b4d5385e2093349e66a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:15.325368 7724 tablet_bootstrap.cc:492] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56: Bootstrap starting.
I20260811 11:50:15.327400 7727 tablet_bootstrap.cc:492] T b4c306d3a03b4d5385e2093349e66a45 P eaf3929d5afa44cdb1aac4e306a3d835: Bootstrap starting.
I20260811 11:50:15.328068 7724 tablet_bootstrap.cc:654] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:15.328176 7728 tablet_bootstrap.cc:492] T b4c306d3a03b4d5385e2093349e66a45 P c4edf677c3f24f6a9b0d7054252810ad: Bootstrap starting.
I20260811 11:50:15.329123 7724 log.cc:824] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:15.329608 7727 tablet_bootstrap.cc:654] T b4c306d3a03b4d5385e2093349e66a45 P eaf3929d5afa44cdb1aac4e306a3d835: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:15.330394 7728 tablet_bootstrap.cc:654] T b4c306d3a03b4d5385e2093349e66a45 P c4edf677c3f24f6a9b0d7054252810ad: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:15.330441 7727 log.cc:824] T b4c306d3a03b4d5385e2093349e66a45 P eaf3929d5afa44cdb1aac4e306a3d835: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:15.331241 7728 log.cc:824] T b4c306d3a03b4d5385e2093349e66a45 P c4edf677c3f24f6a9b0d7054252810ad: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:15.331630 7724 tablet_bootstrap.cc:492] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56: No bootstrap required, opened a new log
I20260811 11:50:15.331844 7724 ts_tablet_manager.cc:1397] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260811 11:50:15.332456 7727 tablet_bootstrap.cc:492] T b4c306d3a03b4d5385e2093349e66a45 P eaf3929d5afa44cdb1aac4e306a3d835: No bootstrap required, opened a new log
I20260811 11:50:15.332656 7727 ts_tablet_manager.cc:1397] T b4c306d3a03b4d5385e2093349e66a45 P eaf3929d5afa44cdb1aac4e306a3d835: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260811 11:50:15.333395 7728 tablet_bootstrap.cc:492] T b4c306d3a03b4d5385e2093349e66a45 P c4edf677c3f24f6a9b0d7054252810ad: No bootstrap required, opened a new log
I20260811 11:50:15.333604 7728 ts_tablet_manager.cc:1397] T b4c306d3a03b4d5385e2093349e66a45 P c4edf677c3f24f6a9b0d7054252810ad: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 11:50:15.335788 7727 raft_consensus.cc:348] T b4c306d3a03b4d5385e2093349e66a45 P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } }
I20260811 11:50:15.336086 7727 raft_consensus.cc:374] T b4c306d3a03b4d5385e2093349e66a45 P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:15.336155 7727 raft_consensus.cc:729] T b4c306d3a03b4d5385e2093349e66a45 P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf3929d5afa44cdb1aac4e306a3d835, State: Initialized, Role: FOLLOWER
I20260811 11:50:15.336619 7727 consensus_queue.cc:260] T b4c306d3a03b4d5385e2093349e66a45 P eaf3929d5afa44cdb1aac4e306a3d835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } }
I20260811 11:50:15.336563 7724 raft_consensus.cc:348] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } }
I20260811 11:50:15.336833 7724 raft_consensus.cc:374] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:15.336894 7724 raft_consensus.cc:729] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de1c7edbf94e40bb8e509194b138cf56, State: Initialized, Role: FOLLOWER
I20260811 11:50:15.337428 7455 heartbeater.cc:503] Master 127.3.221.126:36733 was elected leader, sending a full tablet report...
I20260811 11:50:15.337423 7724 consensus_queue.cc:260] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } }
I20260811 11:50:15.337440 7728 raft_consensus.cc:348] T b4c306d3a03b4d5385e2093349e66a45 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } }
I20260811 11:50:15.337698 7728 raft_consensus.cc:374] T b4c306d3a03b4d5385e2093349e66a45 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:15.337749 7728 raft_consensus.cc:729] T b4c306d3a03b4d5385e2093349e66a45 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4edf677c3f24f6a9b0d7054252810ad, State: Initialized, Role: FOLLOWER
I20260811 11:50:15.337791 7727 ts_tablet_manager.cc:1428] T b4c306d3a03b4d5385e2093349e66a45 P eaf3929d5afa44cdb1aac4e306a3d835: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260811 11:50:15.338169 7728 consensus_queue.cc:260] T b4c306d3a03b4d5385e2093349e66a45 P c4edf677c3f24f6a9b0d7054252810ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } }
I20260811 11:50:15.338479 7724 ts_tablet_manager.cc:1428] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260811 11:50:15.338943 7693 heartbeater.cc:503] Master 127.3.221.126:36733 was elected leader, sending a full tablet report...
I20260811 11:50:15.339100 7221 heartbeater.cc:503] Master 127.3.221.126:36733 was elected leader, sending a full tablet report...
I20260811 11:50:15.339164 7728 ts_tablet_manager.cc:1428] T b4c306d3a03b4d5385e2093349e66a45 P c4edf677c3f24f6a9b0d7054252810ad: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260811 11:50:15.373939 7735 raft_consensus.cc:482] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:50:15.374168 7735 raft_consensus.cc:504] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } }
I20260811 11:50:15.375458 7735 leader_election.cc:288] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4edf677c3f24f6a9b0d7054252810ad (127.3.221.67:43685), eaf3929d5afa44cdb1aac4e306a3d835 (127.3.221.66:45007)
I20260811 11:50:15.379776 7722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.375825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:50425 (local address 127.3.221.67:43685)
0811 11:50:15.375956 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:15.375962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:15.376289 (+ 327us) server_negotiation.cc:407] Connection header received
0811 11:50:15.376508 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:15.376511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:15.376555 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:15.376607 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:15.376933 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.377506 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.378752 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.379185 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.379309 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.379364 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:15.379556 (+ 192us) server_negotiation.cc:299] Negotiation successful
0811 11:50:15.379629 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 11:50:15.379794 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.375711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:43685 (local address 127.3.221.65:50425)
0811 11:50:15.376146 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:15.376159 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:15.376302 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:15.376646 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:15.376649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:15.376667 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:15.376812 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.376818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.377622 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.377626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.378596 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.378607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.379310 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.379313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.379423 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.379458 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 11:50:15.379527 (+ 69us) client_negotiation.cc:240] Negotiation successful
0811 11:50:15.379609 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":83,"spinlock_wait_cycles":22272,"thread_start_us":144,"threads_started":1}
I20260811 11:50:15.379801 7723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.375944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:39897 (local address 127.3.221.66:45007)
0811 11:50:15.376082 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:15.376086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:15.376418 (+ 332us) server_negotiation.cc:407] Connection header received
0811 11:50:15.376521 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:15.376524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:15.376567 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:15.376644 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:15.376947 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.377519 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.378747 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.379173 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.379294 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.379354 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:15.379589 (+ 235us) server_negotiation.cc:299] Negotiation successful
0811 11:50:15.379667 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 11:50:15.380256 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.375791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:45007 (local address 127.3.221.65:39897)
0811 11:50:15.376220 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:15.376232 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:15.376355 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:15.376676 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:15.376679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:15.376721 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:15.376826 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.376831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.377635 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.377639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.378592 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.378603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.379303 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.379306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.379464 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.379483 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 11:50:15.379550 (+ 67us) client_negotiation.cc:240] Negotiation successful
0811 11:50:15.379622 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":40,"spinlock_wait_cycles":768,"thread_start_us":166,"threads_started":1}
I20260811 11:50:15.380539 7606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c306d3a03b4d5385e2093349e66a45" candidate_uuid: "de1c7edbf94e40bb8e509194b138cf56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4edf677c3f24f6a9b0d7054252810ad" is_pre_election: true
I20260811 11:50:15.380919 7606 raft_consensus.cc:2454] T b4c306d3a03b4d5385e2093349e66a45 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de1c7edbf94e40bb8e509194b138cf56 in term 0.
I20260811 11:50:15.381001 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c306d3a03b4d5385e2093349e66a45" candidate_uuid: "de1c7edbf94e40bb8e509194b138cf56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" is_pre_election: true
I20260811 11:50:15.381322 7382 raft_consensus.cc:2454] T b4c306d3a03b4d5385e2093349e66a45 P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de1c7edbf94e40bb8e509194b138cf56 in term 0.
I20260811 11:50:15.381589 7012 leader_election.cc:302] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4edf677c3f24f6a9b0d7054252810ad, de1c7edbf94e40bb8e509194b138cf56; no voters:
I20260811 11:50:15.381932 7735 raft_consensus.cc:2781] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:50:15.382033 7735 raft_consensus.cc:482] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:50:15.382078 7735 raft_consensus.cc:3037] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:15.383164 7735 raft_consensus.cc:504] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } }
I20260811 11:50:15.383543 7735 leader_election.cc:288] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56 [CANDIDATE]: Term 1 election: Requested vote from peers c4edf677c3f24f6a9b0d7054252810ad (127.3.221.67:43685), eaf3929d5afa44cdb1aac4e306a3d835 (127.3.221.66:45007)
I20260811 11:50:15.383958 7606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c306d3a03b4d5385e2093349e66a45" candidate_uuid: "de1c7edbf94e40bb8e509194b138cf56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4edf677c3f24f6a9b0d7054252810ad"
I20260811 11:50:15.383963 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c306d3a03b4d5385e2093349e66a45" candidate_uuid: "de1c7edbf94e40bb8e509194b138cf56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf3929d5afa44cdb1aac4e306a3d835"
I20260811 11:50:15.384104 7382 raft_consensus.cc:3037] T b4c306d3a03b4d5385e2093349e66a45 P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:15.384104 7606 raft_consensus.cc:3037] T b4c306d3a03b4d5385e2093349e66a45 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:15.385550 7606 raft_consensus.cc:2454] T b4c306d3a03b4d5385e2093349e66a45 P c4edf677c3f24f6a9b0d7054252810ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de1c7edbf94e40bb8e509194b138cf56 in term 1.
I20260811 11:50:15.385972 7382 raft_consensus.cc:2454] T b4c306d3a03b4d5385e2093349e66a45 P eaf3929d5afa44cdb1aac4e306a3d835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de1c7edbf94e40bb8e509194b138cf56 in term 1.
I20260811 11:50:15.386000 7012 leader_election.cc:302] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4edf677c3f24f6a9b0d7054252810ad, de1c7edbf94e40bb8e509194b138cf56; no voters:
I20260811 11:50:15.386207 7735 raft_consensus.cc:2781] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:15.386451 7735 raft_consensus.cc:686] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56 [term 1 LEADER]: Becoming Leader. State: Replica: de1c7edbf94e40bb8e509194b138cf56, State: Running, Role: LEADER
I20260811 11:50:15.386802 7735 consensus_queue.cc:237] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } }
I20260811 11:50:15.390372 6865 catalog_manager.cc:5495] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56 reported cstate change: term changed from 0 to 1, leader changed from <none> to de1c7edbf94e40bb8e509194b138cf56 (127.3.221.65). New cstate: current_term: 1 leader_uuid: "de1c7edbf94e40bb8e509194b138cf56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } health_report { overall_health: UNKNOWN } } }
Aug 11 11:50:15 dist-test-slave-bndk krb5kdc[6788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449014, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:15.443694 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.435125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55818 (local address 127.3.221.126:36733)
0811 11:50:15.435262 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:15.435267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:15.435511 (+ 244us) server_negotiation.cc:407] Connection header received
0811 11:50:15.435642 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:15.435645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:15.435685 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:15.435752 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:15.436495 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.437014 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.437830 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.438034 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.438121 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.440495 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:15.440499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:15.440501 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:15.440526 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:15.441865 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:15.442339 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:15.442342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:15.442344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:15.442427 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:15.442748 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:15.442757 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:15.442759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:15.442959 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:15.443035 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:15.443442 (+ 407us) server_negotiation.cc:299] Negotiation successful
0811 11:50:15.443554 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 11:50:15.443701 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.435055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36733 (local address 127.0.0.1:55818)
0811 11:50:15.435386 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:15.435399 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:15.435550 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:15.435795 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:15.435798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:15.436248 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:15.436363 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.436370 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.437127 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.437130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.437705 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.437714 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.438152 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.438155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.438232 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.438822 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:15.438834 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:15.440332 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:15.441976 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:15.441985 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:15.441992 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:15.442224 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:15.442546 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:15.442549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:15.442551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:15.442647 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:15.443097 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:15.443103 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:15.443236 (+ 133us) client_negotiation.cc:769] Sending connection context
0811 11:50:15.443358 (+ 122us) client_negotiation.cc:240] Negotiation successful
0811 11:50:15.443494 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":130,"threads_started":1}
I20260811 11:50:15.447885 6865 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:55818:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 11:50:15.448695 6865 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:50:15.453995 7098 tablet_service.cc:1467] Processing CreateTablet for tablet 6b8c6093524d4deabc5e13c7e428601f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a02979293dc249adb0ce12778a89439d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 11:50:15.454154 7354 tablet_service.cc:1467] Processing CreateTablet for tablet 6b8c6093524d4deabc5e13c7e428601f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a02979293dc249adb0ce12778a89439d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 11:50:15.454383 7580 tablet_service.cc:1467] Processing CreateTablet for tablet 6b8c6093524d4deabc5e13c7e428601f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a02979293dc249adb0ce12778a89439d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 11:50:15.454865 7098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b8c6093524d4deabc5e13c7e428601f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:15.454895 7354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b8c6093524d4deabc5e13c7e428601f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:15.454931 7580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b8c6093524d4deabc5e13c7e428601f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:15.457628 7727 tablet_bootstrap.cc:492] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835: Bootstrap starting.
I20260811 11:50:15.457830 7724 tablet_bootstrap.cc:492] T 6b8c6093524d4deabc5e13c7e428601f P de1c7edbf94e40bb8e509194b138cf56: Bootstrap starting.
I20260811 11:50:15.458146 7728 tablet_bootstrap.cc:492] T 6b8c6093524d4deabc5e13c7e428601f P c4edf677c3f24f6a9b0d7054252810ad: Bootstrap starting.
I20260811 11:50:15.458701 7724 tablet_bootstrap.cc:654] T 6b8c6093524d4deabc5e13c7e428601f P de1c7edbf94e40bb8e509194b138cf56: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:15.458701 7727 tablet_bootstrap.cc:654] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:15.459022 7728 tablet_bootstrap.cc:654] T 6b8c6093524d4deabc5e13c7e428601f P c4edf677c3f24f6a9b0d7054252810ad: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:15.460383 7727 tablet_bootstrap.cc:492] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835: No bootstrap required, opened a new log
I20260811 11:50:15.460451 7728 tablet_bootstrap.cc:492] T 6b8c6093524d4deabc5e13c7e428601f P c4edf677c3f24f6a9b0d7054252810ad: No bootstrap required, opened a new log
I20260811 11:50:15.460470 7727 ts_tablet_manager.cc:1397] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 11:50:15.460523 7728 ts_tablet_manager.cc:1397] T 6b8c6093524d4deabc5e13c7e428601f P c4edf677c3f24f6a9b0d7054252810ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 11:50:15.460892 7727 raft_consensus.cc:348] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:15.460932 7728 raft_consensus.cc:348] T 6b8c6093524d4deabc5e13c7e428601f P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:15.460995 7727 raft_consensus.cc:374] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:15.461020 7728 raft_consensus.cc:374] T 6b8c6093524d4deabc5e13c7e428601f P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:15.461025 7727 raft_consensus.cc:729] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf3929d5afa44cdb1aac4e306a3d835, State: Initialized, Role: FOLLOWER
I20260811 11:50:15.461041 7728 raft_consensus.cc:729] T 6b8c6093524d4deabc5e13c7e428601f P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4edf677c3f24f6a9b0d7054252810ad, State: Initialized, Role: FOLLOWER
I20260811 11:50:15.461158 7727 consensus_queue.cc:260] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:15.461158 7728 consensus_queue.cc:260] T 6b8c6093524d4deabc5e13c7e428601f P c4edf677c3f24f6a9b0d7054252810ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:15.461484 7728 ts_tablet_manager.cc:1428] T 6b8c6093524d4deabc5e13c7e428601f P c4edf677c3f24f6a9b0d7054252810ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:15.461453 7733 tablet_replica.cc:439] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } } }
I20260811 11:50:15.461586 7733 tablet_replica.cc:442] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:50:15.461799 7727 ts_tablet_manager.cc:1428] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:15.462054 7736 tablet_replica.cc:439] T 6b8c6093524d4deabc5e13c7e428601f P c4edf677c3f24f6a9b0d7054252810ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } } }
I20260811 11:50:15.462180 7736 tablet_replica.cc:442] T 6b8c6093524d4deabc5e13c7e428601f P c4edf677c3f24f6a9b0d7054252810ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:50:15.462239 7724 tablet_bootstrap.cc:492] T 6b8c6093524d4deabc5e13c7e428601f P de1c7edbf94e40bb8e509194b138cf56: No bootstrap required, opened a new log
I20260811 11:50:15.462309 7724 ts_tablet_manager.cc:1397] T 6b8c6093524d4deabc5e13c7e428601f P de1c7edbf94e40bb8e509194b138cf56: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260811 11:50:15.462785 7724 raft_consensus.cc:348] T 6b8c6093524d4deabc5e13c7e428601f P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:15.462890 7724 raft_consensus.cc:374] T 6b8c6093524d4deabc5e13c7e428601f P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:15.462921 7724 raft_consensus.cc:729] T 6b8c6093524d4deabc5e13c7e428601f P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de1c7edbf94e40bb8e509194b138cf56, State: Initialized, Role: FOLLOWER
I20260811 11:50:15.463055 7724 consensus_queue.cc:260] T 6b8c6093524d4deabc5e13c7e428601f P de1c7edbf94e40bb8e509194b138cf56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:15.463315 7724 ts_tablet_manager.cc:1428] T 6b8c6093524d4deabc5e13c7e428601f P de1c7edbf94e40bb8e509194b138cf56: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260811 11:50:15.463344 7735 tablet_replica.cc:439] T 6b8c6093524d4deabc5e13c7e428601f P de1c7edbf94e40bb8e509194b138cf56: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } } }
I20260811 11:50:15.463476 7735 tablet_replica.cc:442] T 6b8c6093524d4deabc5e13c7e428601f P de1c7edbf94e40bb8e509194b138cf56: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:50:15.481513 7733 raft_consensus.cc:482] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:50:15.481654 7733 raft_consensus.cc:504] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:15.482865 7733 leader_election.cc:288] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de1c7edbf94e40bb8e509194b138cf56 (127.3.221.65:41349), c4edf677c3f24f6a9b0d7054252810ad (127.3.221.67:43685)
I20260811 11:50:15.486891 7722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.483331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:42243 (local address 127.3.221.67:43685)
0811 11:50:15.483444 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:15.483449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:15.483700 (+ 251us) server_negotiation.cc:407] Connection header received
0811 11:50:15.483827 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:15.483830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:15.483875 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:15.483952 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:15.484236 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.484709 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.485815 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.486267 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.486398 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.486451 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:15.486662 (+ 211us) server_negotiation.cc:299] Negotiation successful
0811 11:50:15.486744 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 11:50:15.486924 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.483115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:41349 (local address 127.3.221.66:35383)
0811 11:50:15.483411 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:15.483425 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:15.483527 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:15.483810 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:15.483812 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:15.483832 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:15.483964 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.483971 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.484766 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.484784 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.485656 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.485666 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.486533 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.486537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.486639 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.486663 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 11:50:15.486737 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 11:50:15.486812 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":3}
I20260811 11:50:15.486950 7719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.483243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:35383 (local address 127.3.221.65:41349)
0811 11:50:15.483403 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:15.483408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:15.483487 (+ 79us) server_negotiation.cc:407] Connection header received
0811 11:50:15.483646 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:15.483649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:15.483702 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:15.483780 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:15.484094 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.484653 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.485785 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.486335 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.486459 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.486511 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:15.486774 (+ 263us) server_negotiation.cc:299] Negotiation successful
0811 11:50:15.486846 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 11:50:15.487247 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.483240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:43685 (local address 127.3.221.66:42243)
0811 11:50:15.483563 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:15.483576 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:15.483715 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:15.483976 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:15.483978 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:15.483993 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:15.484130 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.484136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.484823 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.484826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.485685 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.485695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.486404 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.486408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.486498 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.486533 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 11:50:15.487048 (+ 515us) client_negotiation.cc:240] Negotiation successful
0811 11:50:15.487125 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":142,"threads_started":1}
I20260811 11:50:15.487552 7133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b8c6093524d4deabc5e13c7e428601f" candidate_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de1c7edbf94e40bb8e509194b138cf56" is_pre_election: true
I20260811 11:50:15.487812 7133 raft_consensus.cc:2454] T 6b8c6093524d4deabc5e13c7e428601f P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf3929d5afa44cdb1aac4e306a3d835 in term 0.
I20260811 11:50:15.488137 7606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b8c6093524d4deabc5e13c7e428601f" candidate_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4edf677c3f24f6a9b0d7054252810ad" is_pre_election: true
I20260811 11:50:15.488266 7606 raft_consensus.cc:2454] T 6b8c6093524d4deabc5e13c7e428601f P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf3929d5afa44cdb1aac4e306a3d835 in term 0.
I20260811 11:50:15.488242 7269 leader_election.cc:302] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de1c7edbf94e40bb8e509194b138cf56, eaf3929d5afa44cdb1aac4e306a3d835; no voters:
I20260811 11:50:15.488546 7733 raft_consensus.cc:2781] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:50:15.488612 7733 raft_consensus.cc:482] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:50:15.488656 7733 raft_consensus.cc:3037] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:15.489710 7733 raft_consensus.cc:504] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:15.490058 7733 leader_election.cc:288] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835 [CANDIDATE]: Term 1 election: Requested vote from peers de1c7edbf94e40bb8e509194b138cf56 (127.3.221.65:41349), c4edf677c3f24f6a9b0d7054252810ad (127.3.221.67:43685)
I20260811 11:50:15.490423 7133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b8c6093524d4deabc5e13c7e428601f" candidate_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de1c7edbf94e40bb8e509194b138cf56"
I20260811 11:50:15.490554 7133 raft_consensus.cc:3037] T 6b8c6093524d4deabc5e13c7e428601f P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:15.490635 7606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b8c6093524d4deabc5e13c7e428601f" candidate_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4edf677c3f24f6a9b0d7054252810ad"
I20260811 11:50:15.490770 7606 raft_consensus.cc:3037] T 6b8c6093524d4deabc5e13c7e428601f P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:15.491544 7133 raft_consensus.cc:2454] T 6b8c6093524d4deabc5e13c7e428601f P de1c7edbf94e40bb8e509194b138cf56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf3929d5afa44cdb1aac4e306a3d835 in term 1.
I20260811 11:50:15.491719 7606 raft_consensus.cc:2454] T 6b8c6093524d4deabc5e13c7e428601f P c4edf677c3f24f6a9b0d7054252810ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf3929d5afa44cdb1aac4e306a3d835 in term 1.
I20260811 11:50:15.491835 7269 leader_election.cc:302] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de1c7edbf94e40bb8e509194b138cf56, eaf3929d5afa44cdb1aac4e306a3d835; no voters:
I20260811 11:50:15.492009 7733 raft_consensus.cc:2781] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:15.492290 7733 raft_consensus.cc:686] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835 [term 1 LEADER]: Becoming Leader. State: Replica: eaf3929d5afa44cdb1aac4e306a3d835, State: Running, Role: LEADER
I20260811 11:50:15.492631 7733 consensus_queue.cc:237] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:15.495764 7755 tablet_replica.cc:439] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835: TxnStatusTablet state changed. Reason: New leader eaf3929d5afa44cdb1aac4e306a3d835. Latest consensus state: current_term: 1 leader_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } } }
I20260811 11:50:15.495879 7755 tablet_replica.cc:442] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835: This TxnStatusTablet replica's current role is: LEADER
I20260811 11:50:15.495991 6865 catalog_manager.cc:5495] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf3929d5afa44cdb1aac4e306a3d835 (127.3.221.66). New cstate: current_term: 1 leader_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } health_report { overall_health: UNKNOWN } } }
I20260811 11:50:15.496157 7759 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 11:50:15.496358 7759 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260811 11:50:15.504925 7694 tablet.cc:2369] T 6b8c6093524d4deabc5e13c7e428601f P c4edf677c3f24f6a9b0d7054252810ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:50:15.515872 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.511521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:45007 (local address 127.0.0.1:54112)
0811 11:50:15.511698 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:15.511714 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:15.511811 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:15.512166 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:15.512168 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:15.512174 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:50:15.512274 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.512279 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.513309 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.513312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.513928 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.513938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.514544 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.514547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.514628 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.514680 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:50:15.515568 (+ 888us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:50:15.515587 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 11:50:15.515658 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 11:50:15.515743 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260811 11:50:15.515918 7723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.511626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54112 (local address 127.3.221.66:45007)
0811 11:50:15.511763 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:15.511767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:15.511785 (+ 18us) server_negotiation.cc:407] Connection header received
0811 11:50:15.511967 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:15.511970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:15.512036 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:15.512156 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:15.512376 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.513187 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.514048 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.514392 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.514497 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.514829 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:15.515440 (+ 611us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:15.515557 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:15.515697 (+ 140us) server_negotiation.cc:299] Negotiation successful
0811 11:50:15.515756 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 11:50:15.521917 7133 raft_consensus.cc:1261] T 6b8c6093524d4deabc5e13c7e428601f P de1c7edbf94e40bb8e509194b138cf56 [term 1 FOLLOWER]: Refusing update from remote peer eaf3929d5afa44cdb1aac4e306a3d835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:15.522014 7606 raft_consensus.cc:1261] T 6b8c6093524d4deabc5e13c7e428601f P c4edf677c3f24f6a9b0d7054252810ad [term 1 FOLLOWER]: Refusing update from remote peer eaf3929d5afa44cdb1aac4e306a3d835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:15.522569 7755 consensus_queue.cc:1035] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:15.522781 7733 consensus_queue.cc:1035] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 11:50:15.525604 7457 tablet.cc:2369] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:50:15.526091 7735 tablet_replica.cc:439] T 6b8c6093524d4deabc5e13c7e428601f P de1c7edbf94e40bb8e509194b138cf56: TxnStatusTablet state changed. Reason: New leader eaf3929d5afa44cdb1aac4e306a3d835. Latest consensus state: current_term: 1 leader_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } } }
I20260811 11:50:15.526201 7735 tablet_replica.cc:442] T 6b8c6093524d4deabc5e13c7e428601f P de1c7edbf94e40bb8e509194b138cf56: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:50:15.528272 7733 tablet_replica.cc:439] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } } }
I20260811 11:50:15.528393 7733 tablet_replica.cc:442] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835: This TxnStatusTablet replica's current role is: LEADER
I20260811 11:50:15.528538 7736 tablet_replica.cc:439] T 6b8c6093524d4deabc5e13c7e428601f P c4edf677c3f24f6a9b0d7054252810ad: TxnStatusTablet state changed. Reason: New leader eaf3929d5afa44cdb1aac4e306a3d835. Latest consensus state: current_term: 1 leader_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } } }
I20260811 11:50:15.528659 7736 tablet_replica.cc:442] T 6b8c6093524d4deabc5e13c7e428601f P c4edf677c3f24f6a9b0d7054252810ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:50:15.529808 7735 tablet_replica.cc:439] T 6b8c6093524d4deabc5e13c7e428601f P de1c7edbf94e40bb8e509194b138cf56: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } } }
I20260811 11:50:15.529934 7735 tablet_replica.cc:442] T 6b8c6093524d4deabc5e13c7e428601f P de1c7edbf94e40bb8e509194b138cf56: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:50:15.530296 7763 mvcc.cc:205] Tried to move back new op lower bound from 7317295167570165760 to 7317295167460937728. Current Snapshot: MvccSnapshot[applied={T|T < 7317295167570165760}]
I20260811 11:50:15.531543 7766 mvcc.cc:205] Tried to move back new op lower bound from 7317295167570165760 to 7317295167460937728. Current Snapshot: MvccSnapshot[applied={T|T < 7317295167570165760}]
W20260811 11:50:15.535676 7222 tablet.cc:2369] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:50:15.537781 7755 tablet_replica.cc:439] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } } }
I20260811 11:50:15.537931 7755 tablet_replica.cc:442] T 6b8c6093524d4deabc5e13c7e428601f P eaf3929d5afa44cdb1aac4e306a3d835: This TxnStatusTablet replica's current role is: LEADER
I20260811 11:50:15.540416 7736 tablet_replica.cc:439] T 6b8c6093524d4deabc5e13c7e428601f P c4edf677c3f24f6a9b0d7054252810ad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } } }
I20260811 11:50:15.540553 7736 tablet_replica.cc:442] T 6b8c6093524d4deabc5e13c7e428601f P c4edf677c3f24f6a9b0d7054252810ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:50:15.541145 7764 mvcc.cc:205] Tried to move back new op lower bound from 7317295167570165760 to 7317295167460937728. Current Snapshot: MvccSnapshot[applied={T|T < 7317295167570165760}]
I20260811 11:50:15.543553 7719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.537629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.3.221.65:41349)
0811 11:50:15.537740 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:15.537745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:15.537811 (+ 66us) server_negotiation.cc:407] Connection header received
0811 11:50:15.538010 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:15.538013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:15.538077 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:15.538172 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:15.538467 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.539163 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.541323 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.541633 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.541745 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.542312 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:15.542744 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:15.542862 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:15.543329 (+ 467us) server_negotiation.cc:299] Negotiation successful
0811 11:50:15.543405 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260811 11:50:15.555594 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.551776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:45007 (local address 127.3.221.65:36059)
0811 11:50:15.552105 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:15.552118 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:15.552224 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:15.552471 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:15.552473 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:15.552478 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:50:15.552593 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.552598 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.553649 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.553652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.554317 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.554327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.554890 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.554893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.554968 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.555018 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:50:15.555330 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:50:15.555335 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 11:50:15.555382 (+ 47us) client_negotiation.cc:240] Negotiation successful
0811 11:50:15.555449 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":112,"threads_started":1}
I20260811 11:50:15.555588 7723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.551902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:36059 (local address 127.3.221.66:45007)
0811 11:50:15.552031 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:15.552036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:15.552194 (+ 158us) server_negotiation.cc:407] Connection header received
0811 11:50:15.552314 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:15.552318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:15.552374 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:15.552454 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:15.552681 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.553512 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.554433 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.554777 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.554900 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.555166 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:15.555251 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:15.555322 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:15.555396 (+ 74us) server_negotiation.cc:299] Negotiation successful
0811 11:50:15.555453 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 11:50:15.563294 7606 raft_consensus.cc:1261] T b4c306d3a03b4d5385e2093349e66a45 P c4edf677c3f24f6a9b0d7054252810ad [term 1 FOLLOWER]: Refusing update from remote peer de1c7edbf94e40bb8e509194b138cf56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:15.563549 7382 raft_consensus.cc:1261] T b4c306d3a03b4d5385e2093349e66a45 P eaf3929d5afa44cdb1aac4e306a3d835 [term 1 FOLLOWER]: Refusing update from remote peer de1c7edbf94e40bb8e509194b138cf56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:15.563905 7740 consensus_queue.cc:1035] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:15.564087 7735 consensus_queue.cc:1035] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:15.590273 7719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.585896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:41205 (local address 127.3.221.65:41349)
0811 11:50:15.586040 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:15.586045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:15.586342 (+ 297us) server_negotiation.cc:407] Connection header received
0811 11:50:15.586523 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:15.586527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:15.586583 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:15.586666 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:15.586976 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.587751 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.588998 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.589345 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.589434 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.589802 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:15.589906 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:15.589976 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:15.590064 (+ 88us) server_negotiation.cc:299] Negotiation successful
0811 11:50:15.590124 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 11:50:15.590651 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:15.585798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:41349 (local address 127.3.221.66:41205)
0811 11:50:15.586226 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:15.586239 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:15.586371 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:15.586694 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:15.586697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:15.586706 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:50:15.586854 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.586862 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.587911 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.587914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.588887 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:15.588899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:15.589453 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:15.589456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:15.589559 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:15.589622 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:50:15.589985 (+ 363us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:50:15.589992 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 11:50:15.590055 (+ 63us) client_negotiation.cc:240] Negotiation successful
0811 11:50:15.590139 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":133,"threads_started":1}
I20260811 11:50:15.614820 6860 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:15.615051 6860 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:15.618047 6860 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a: Sending DeleteTablet for 3 replicas of tablet b4c306d3a03b4d5385e2093349e66a45
I20260811 11:50:15.619021 7580 tablet_service.cc:1514] Processing DeleteTablet for tablet b4c306d3a03b4d5385e2093349e66a45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:15 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:49786
I20260811 11:50:15.619462 7098 tablet_service.cc:1514] Processing DeleteTablet for tablet b4c306d3a03b4d5385e2093349e66a45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:15 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:39378
I20260811 11:50:15.619992 7799 tablet_replica.cc:330] T b4c306d3a03b4d5385e2093349e66a45 P c4edf677c3f24f6a9b0d7054252810ad: stopping tablet replica
I20260811 11:50:15.619971 7354 tablet_service.cc:1514] Processing DeleteTablet for tablet b4c306d3a03b4d5385e2093349e66a45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:15 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:54104
I20260811 11:50:15.620458 7799 raft_consensus.cc:2229] T b4c306d3a03b4d5385e2093349e66a45 P c4edf677c3f24f6a9b0d7054252810ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:15.620787 7799 raft_consensus.cc:2258] T b4c306d3a03b4d5385e2093349e66a45 P c4edf677c3f24f6a9b0d7054252810ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:15.621359 6860 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:50:15.621860 6860 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:50:15.622776 7802 tablet_replica.cc:330] T b4c306d3a03b4d5385e2093349e66a45 P eaf3929d5afa44cdb1aac4e306a3d835: stopping tablet replica
I20260811 11:50:15.623107 7802 raft_consensus.cc:2229] T b4c306d3a03b4d5385e2093349e66a45 P eaf3929d5afa44cdb1aac4e306a3d835 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:15.623283 7802 raft_consensus.cc:2258] T b4c306d3a03b4d5385e2093349e66a45 P eaf3929d5afa44cdb1aac4e306a3d835 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:15.624827 7802 ts_tablet_manager.cc:1905] T b4c306d3a03b4d5385e2093349e66a45 P eaf3929d5afa44cdb1aac4e306a3d835: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:15.628209 7580 tablet_service.cc:1467] Processing CreateTablet for tablet 677f5b4668434e189b43a9e4671864f8 (DEFAULT_TABLE table=test-table [id=98b0ef963b3b441f968eabd58b020256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:15.628326 7098 tablet_service.cc:1467] Processing CreateTablet for tablet 677f5b4668434e189b43a9e4671864f8 (DEFAULT_TABLE table=test-table [id=98b0ef963b3b441f968eabd58b020256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:15.628553 7580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 677f5b4668434e189b43a9e4671864f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:15.628634 7098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 677f5b4668434e189b43a9e4671864f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:15.629451 7801 tablet_replica.cc:330] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56: stopping tablet replica
I20260811 11:50:15.629798 7801 raft_consensus.cc:2229] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:50:15.630200 7801 raft_consensus.cc:2258] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:15.631422 7728 tablet_bootstrap.cc:492] T 677f5b4668434e189b43a9e4671864f8 P c4edf677c3f24f6a9b0d7054252810ad: Bootstrap starting.
I20260811 11:50:15.631438 7354 tablet_service.cc:1467] Processing CreateTablet for tablet 677f5b4668434e189b43a9e4671864f8 (DEFAULT_TABLE table=test-table [id=98b0ef963b3b441f968eabd58b020256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:15.631732 7354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 677f5b4668434e189b43a9e4671864f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:15.631784 7801 ts_tablet_manager.cc:1905] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:15.632407 7728 tablet_bootstrap.cc:654] T 677f5b4668434e189b43a9e4671864f8 P c4edf677c3f24f6a9b0d7054252810ad: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:15.633843 7799 ts_tablet_manager.cc:1905] T b4c306d3a03b4d5385e2093349e66a45 P c4edf677c3f24f6a9b0d7054252810ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:15.634754 7801 ts_tablet_manager.cc:1918] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:50:15.634867 7801 log.cc:1196] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-0/wal/wals/b4c306d3a03b4d5385e2093349e66a45
I20260811 11:50:15.635185 7801 ts_tablet_manager.cc:1939] T b4c306d3a03b4d5385e2093349e66a45 P de1c7edbf94e40bb8e509194b138cf56: Deleting consensus metadata
I20260811 11:50:15.635505 7727 tablet_bootstrap.cc:492] T 677f5b4668434e189b43a9e4671864f8 P eaf3929d5afa44cdb1aac4e306a3d835: Bootstrap starting.
I20260811 11:50:15.636565 7727 tablet_bootstrap.cc:654] T 677f5b4668434e189b43a9e4671864f8 P eaf3929d5afa44cdb1aac4e306a3d835: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:15.636557 6840 catalog_manager.cc:4841] TS de1c7edbf94e40bb8e509194b138cf56 (127.3.221.65:41349): tablet b4c306d3a03b4d5385e2093349e66a45 (table test-table [id=0a82454f45914818a503624a2cbe0459]) successfully deleted
I20260811 11:50:15.637250 7799 ts_tablet_manager.cc:1918] T b4c306d3a03b4d5385e2093349e66a45 P c4edf677c3f24f6a9b0d7054252810ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:50:15.637428 7799 log.cc:1196] T b4c306d3a03b4d5385e2093349e66a45 P c4edf677c3f24f6a9b0d7054252810ad: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/b4c306d3a03b4d5385e2093349e66a45
I20260811 11:50:15.637811 7799 ts_tablet_manager.cc:1939] T b4c306d3a03b4d5385e2093349e66a45 P c4edf677c3f24f6a9b0d7054252810ad: Deleting consensus metadata
I20260811 11:50:15.638772 7724 tablet_bootstrap.cc:492] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56: Bootstrap starting.
I20260811 11:50:15.639304 7728 tablet_bootstrap.cc:492] T 677f5b4668434e189b43a9e4671864f8 P c4edf677c3f24f6a9b0d7054252810ad: No bootstrap required, opened a new log
I20260811 11:50:15.639397 7728 ts_tablet_manager.cc:1397] T 677f5b4668434e189b43a9e4671864f8 P c4edf677c3f24f6a9b0d7054252810ad: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260811 11:50:15.639853 7724 tablet_bootstrap.cc:654] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:15.640137 7728 raft_consensus.cc:348] T 677f5b4668434e189b43a9e4671864f8 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:15.640254 7728 raft_consensus.cc:374] T 677f5b4668434e189b43a9e4671864f8 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:15.640285 7728 raft_consensus.cc:729] T 677f5b4668434e189b43a9e4671864f8 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4edf677c3f24f6a9b0d7054252810ad, State: Initialized, Role: FOLLOWER
I20260811 11:50:15.640412 7728 consensus_queue.cc:260] T 677f5b4668434e189b43a9e4671864f8 P c4edf677c3f24f6a9b0d7054252810ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:15.640663 7728 ts_tablet_manager.cc:1428] T 677f5b4668434e189b43a9e4671864f8 P c4edf677c3f24f6a9b0d7054252810ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:15.642105 6844 catalog_manager.cc:4841] TS c4edf677c3f24f6a9b0d7054252810ad (127.3.221.67:43685): tablet b4c306d3a03b4d5385e2093349e66a45 (table test-table [id=0a82454f45914818a503624a2cbe0459]) successfully deleted
I20260811 11:50:15.645797 7802 ts_tablet_manager.cc:1918] T b4c306d3a03b4d5385e2093349e66a45 P eaf3929d5afa44cdb1aac4e306a3d835: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:50:15.645933 7802 log.cc:1196] T b4c306d3a03b4d5385e2093349e66a45 P eaf3929d5afa44cdb1aac4e306a3d835: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/b4c306d3a03b4d5385e2093349e66a45
I20260811 11:50:15.646282 7802 ts_tablet_manager.cc:1939] T b4c306d3a03b4d5385e2093349e66a45 P eaf3929d5afa44cdb1aac4e306a3d835: Deleting consensus metadata
I20260811 11:50:15.647778 6845 catalog_manager.cc:4841] TS eaf3929d5afa44cdb1aac4e306a3d835 (127.3.221.66:45007): tablet b4c306d3a03b4d5385e2093349e66a45 (table test-table [id=0a82454f45914818a503624a2cbe0459]) successfully deleted
I20260811 11:50:15.648417 7724 tablet_bootstrap.cc:492] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56: No bootstrap required, opened a new log
I20260811 11:50:15.648535 7724 ts_tablet_manager.cc:1397] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260811 11:50:15.649053 7724 raft_consensus.cc:348] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:15.649189 7724 raft_consensus.cc:374] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:15.649567 7724 raft_consensus.cc:729] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de1c7edbf94e40bb8e509194b138cf56, State: Initialized, Role: FOLLOWER
I20260811 11:50:15.649716 7724 consensus_queue.cc:260] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:15.650036 7724 ts_tablet_manager.cc:1428] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260811 11:50:15.651674 7727 tablet_bootstrap.cc:492] T 677f5b4668434e189b43a9e4671864f8 P eaf3929d5afa44cdb1aac4e306a3d835: No bootstrap required, opened a new log
I20260811 11:50:15.651783 7727 ts_tablet_manager.cc:1397] T 677f5b4668434e189b43a9e4671864f8 P eaf3929d5afa44cdb1aac4e306a3d835: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20260811 11:50:15.652269 7727 raft_consensus.cc:348] T 677f5b4668434e189b43a9e4671864f8 P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:15.652410 7727 raft_consensus.cc:374] T 677f5b4668434e189b43a9e4671864f8 P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:15.652456 7727 raft_consensus.cc:729] T 677f5b4668434e189b43a9e4671864f8 P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf3929d5afa44cdb1aac4e306a3d835, State: Initialized, Role: FOLLOWER
I20260811 11:50:15.652596 7727 consensus_queue.cc:260] T 677f5b4668434e189b43a9e4671864f8 P eaf3929d5afa44cdb1aac4e306a3d835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:15.652895 7727 ts_tablet_manager.cc:1428] T 677f5b4668434e189b43a9e4671864f8 P eaf3929d5afa44cdb1aac4e306a3d835: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:15.938140 7740 raft_consensus.cc:482] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:50:15.938301 7740 raft_consensus.cc:504] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:15.938742 7740 leader_election.cc:288] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf3929d5afa44cdb1aac4e306a3d835 (127.3.221.66:45007), c4edf677c3f24f6a9b0d7054252810ad (127.3.221.67:43685)
I20260811 11:50:15.939244 7605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677f5b4668434e189b43a9e4671864f8" candidate_uuid: "de1c7edbf94e40bb8e509194b138cf56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4edf677c3f24f6a9b0d7054252810ad" is_pre_election: true
I20260811 11:50:15.939242 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677f5b4668434e189b43a9e4671864f8" candidate_uuid: "de1c7edbf94e40bb8e509194b138cf56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" is_pre_election: true
I20260811 11:50:15.939414 7605 raft_consensus.cc:2454] T 677f5b4668434e189b43a9e4671864f8 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de1c7edbf94e40bb8e509194b138cf56 in term 0.
I20260811 11:50:15.939414 7382 raft_consensus.cc:2454] T 677f5b4668434e189b43a9e4671864f8 P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de1c7edbf94e40bb8e509194b138cf56 in term 0.
I20260811 11:50:15.939774 7012 leader_election.cc:302] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4edf677c3f24f6a9b0d7054252810ad, de1c7edbf94e40bb8e509194b138cf56; no voters:
I20260811 11:50:15.939975 7740 raft_consensus.cc:2781] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:50:15.940037 7740 raft_consensus.cc:482] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:50:15.940062 7740 raft_consensus.cc:3037] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:15.941119 7740 raft_consensus.cc:504] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:15.941493 7740 leader_election.cc:288] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56 [CANDIDATE]: Term 1 election: Requested vote from peers eaf3929d5afa44cdb1aac4e306a3d835 (127.3.221.66:45007), c4edf677c3f24f6a9b0d7054252810ad (127.3.221.67:43685)
I20260811 11:50:15.941876 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677f5b4668434e189b43a9e4671864f8" candidate_uuid: "de1c7edbf94e40bb8e509194b138cf56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf3929d5afa44cdb1aac4e306a3d835"
I20260811 11:50:15.941905 7605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677f5b4668434e189b43a9e4671864f8" candidate_uuid: "de1c7edbf94e40bb8e509194b138cf56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4edf677c3f24f6a9b0d7054252810ad"
I20260811 11:50:15.942024 7382 raft_consensus.cc:3037] T 677f5b4668434e189b43a9e4671864f8 P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:15.942029 7605 raft_consensus.cc:3037] T 677f5b4668434e189b43a9e4671864f8 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:15.942942 7382 raft_consensus.cc:2454] T 677f5b4668434e189b43a9e4671864f8 P eaf3929d5afa44cdb1aac4e306a3d835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de1c7edbf94e40bb8e509194b138cf56 in term 1.
I20260811 11:50:15.943022 7605 raft_consensus.cc:2454] T 677f5b4668434e189b43a9e4671864f8 P c4edf677c3f24f6a9b0d7054252810ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de1c7edbf94e40bb8e509194b138cf56 in term 1.
I20260811 11:50:15.943228 7013 leader_election.cc:302] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de1c7edbf94e40bb8e509194b138cf56, eaf3929d5afa44cdb1aac4e306a3d835; no voters:
I20260811 11:50:15.943388 7740 raft_consensus.cc:2781] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:15.943480 7740 raft_consensus.cc:686] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56 [term 1 LEADER]: Becoming Leader. State: Replica: de1c7edbf94e40bb8e509194b138cf56, State: Running, Role: LEADER
I20260811 11:50:15.943631 7740 consensus_queue.cc:237] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:15.945596 6865 catalog_manager.cc:5495] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56 reported cstate change: term changed from 0 to 1, leader changed from <none> to de1c7edbf94e40bb8e509194b138cf56 (127.3.221.65). New cstate: current_term: 1 leader_uuid: "de1c7edbf94e40bb8e509194b138cf56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } health_report { overall_health: UNKNOWN } } }
I20260811 11:50:16.016810 7605 raft_consensus.cc:1261] T 677f5b4668434e189b43a9e4671864f8 P c4edf677c3f24f6a9b0d7054252810ad [term 1 FOLLOWER]: Refusing update from remote peer de1c7edbf94e40bb8e509194b138cf56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:16.016882 7382 raft_consensus.cc:1261] T 677f5b4668434e189b43a9e4671864f8 P eaf3929d5afa44cdb1aac4e306a3d835 [term 1 FOLLOWER]: Refusing update from remote peer de1c7edbf94e40bb8e509194b138cf56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:16.017279 7735 consensus_queue.cc:1035] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:16.017467 7740 consensus_queue.cc:1035] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 11:50:16.032474 7766 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260811 11:50:17.586470 6865 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:17.586637 6865 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:17.589123 6865 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a: Sending DeleteTablet for 3 replicas of tablet 677f5b4668434e189b43a9e4671864f8
I20260811 11:50:17.589818 7353 tablet_service.cc:1514] Processing DeleteTablet for tablet 677f5b4668434e189b43a9e4671864f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:17 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:54104
I20260811 11:50:17.589907 7098 tablet_service.cc:1514] Processing DeleteTablet for tablet 677f5b4668434e189b43a9e4671864f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:17 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:39378
I20260811 11:50:17.590008 7580 tablet_service.cc:1514] Processing DeleteTablet for tablet 677f5b4668434e189b43a9e4671864f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:17 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:49786
I20260811 11:50:17.590220 7856 tablet_replica.cc:330] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56: stopping tablet replica
I20260811 11:50:17.590377 7854 tablet_replica.cc:330] T 677f5b4668434e189b43a9e4671864f8 P eaf3929d5afa44cdb1aac4e306a3d835: stopping tablet replica
I20260811 11:50:17.590394 7856 raft_consensus.cc:2229] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:50:17.590547 7854 raft_consensus.cc:2229] T 677f5b4668434e189b43a9e4671864f8 P eaf3929d5afa44cdb1aac4e306a3d835 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:17.590598 7856 raft_consensus.cc:2258] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:17.590741 7854 raft_consensus.cc:2258] T 677f5b4668434e189b43a9e4671864f8 P eaf3929d5afa44cdb1aac4e306a3d835 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:17.591485 7854 ts_tablet_manager.cc:1905] T 677f5b4668434e189b43a9e4671864f8 P eaf3929d5afa44cdb1aac4e306a3d835: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:17.591595 7856 ts_tablet_manager.cc:1905] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:17.594275 7854 ts_tablet_manager.cc:1918] T 677f5b4668434e189b43a9e4671864f8 P eaf3929d5afa44cdb1aac4e306a3d835: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 11:50:17.594275 7856 ts_tablet_manager.cc:1918] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 11:50:17.594367 7856 log.cc:1196] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-0/wal/wals/677f5b4668434e189b43a9e4671864f8
I20260811 11:50:17.594367 7854 log.cc:1196] T 677f5b4668434e189b43a9e4671864f8 P eaf3929d5afa44cdb1aac4e306a3d835: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/677f5b4668434e189b43a9e4671864f8
I20260811 11:50:17.594623 7854 ts_tablet_manager.cc:1939] T 677f5b4668434e189b43a9e4671864f8 P eaf3929d5afa44cdb1aac4e306a3d835: Deleting consensus metadata
I20260811 11:50:17.594640 7856 ts_tablet_manager.cc:1939] T 677f5b4668434e189b43a9e4671864f8 P de1c7edbf94e40bb8e509194b138cf56: Deleting consensus metadata
I20260811 11:50:17.594858 7857 tablet_replica.cc:330] T 677f5b4668434e189b43a9e4671864f8 P c4edf677c3f24f6a9b0d7054252810ad: stopping tablet replica
I20260811 11:50:17.594991 7857 raft_consensus.cc:2229] T 677f5b4668434e189b43a9e4671864f8 P c4edf677c3f24f6a9b0d7054252810ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:17.595134 7857 raft_consensus.cc:2258] T 677f5b4668434e189b43a9e4671864f8 P c4edf677c3f24f6a9b0d7054252810ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:17.595639 6840 catalog_manager.cc:4841] TS de1c7edbf94e40bb8e509194b138cf56 (127.3.221.65:41349): tablet 677f5b4668434e189b43a9e4671864f8 (table test-table [id=98b0ef963b3b441f968eabd58b020256]) successfully deleted
I20260811 11:50:17.595799 7857 ts_tablet_manager.cc:1905] T 677f5b4668434e189b43a9e4671864f8 P c4edf677c3f24f6a9b0d7054252810ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:17.596065 6845 catalog_manager.cc:4841] TS eaf3929d5afa44cdb1aac4e306a3d835 (127.3.221.66:45007): tablet 677f5b4668434e189b43a9e4671864f8 (table test-table [id=98b0ef963b3b441f968eabd58b020256]) successfully deleted
I20260811 11:50:17.597941 7857 ts_tablet_manager.cc:1918] T 677f5b4668434e189b43a9e4671864f8 P c4edf677c3f24f6a9b0d7054252810ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 11:50:17.598009 7857 log.cc:1196] T 677f5b4668434e189b43a9e4671864f8 P c4edf677c3f24f6a9b0d7054252810ad: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/677f5b4668434e189b43a9e4671864f8
I20260811 11:50:17.598241 7857 ts_tablet_manager.cc:1939] T 677f5b4668434e189b43a9e4671864f8 P c4edf677c3f24f6a9b0d7054252810ad: Deleting consensus metadata
I20260811 11:50:17.598879 6844 catalog_manager.cc:4841] TS c4edf677c3f24f6a9b0d7054252810ad (127.3.221.67:43685): tablet 677f5b4668434e189b43a9e4671864f8 (table test-table [id=98b0ef963b3b441f968eabd58b020256]) successfully deleted
Aug 11 11:50:17 dist-test-slave-bndk krb5kdc[6788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449017, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 11:50:17 dist-test-slave-bndk krb5kdc[6788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449017, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:17.618705 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:17.609499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47356 (local address 127.3.221.126:36733)
0811 11:50:17.609772 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:17.609779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:17.609995 (+ 216us) server_negotiation.cc:407] Connection header received
0811 11:50:17.610123 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:17.610126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:17.610172 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:17.610254 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:17.611077 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:17.611731 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:17.612575 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:17.612862 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:17.612943 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:17.615540 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:17.615543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:17.615545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:17.615574 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:17.616897 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:17.617377 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:17.617381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:17.617383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:17.617476 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:17.617765 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:17.617768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:17.617769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:17.617962 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:17.618087 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:17.618399 (+ 312us) server_negotiation.cc:299] Negotiation successful
0811 11:50:17.618534 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":91,"threads_started":1}
I20260811 11:50:17.621775 6865 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:50:17.622295 6865 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:50:17.627282 7098 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4d807ae5bc44ef84eb5b451463da51 (DEFAULT_TABLE table=test-table [id=0a6607882bec4b9fa0e666817d596cd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:17.627282 7353 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4d807ae5bc44ef84eb5b451463da51 (DEFAULT_TABLE table=test-table [id=0a6607882bec4b9fa0e666817d596cd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:17.627332 7580 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4d807ae5bc44ef84eb5b451463da51 (DEFAULT_TABLE table=test-table [id=0a6607882bec4b9fa0e666817d596cd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:17.627620 7098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b4d807ae5bc44ef84eb5b451463da51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:17.627620 7353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b4d807ae5bc44ef84eb5b451463da51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:17.627651 7580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b4d807ae5bc44ef84eb5b451463da51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:17.629843 7875 tablet_bootstrap.cc:492] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad: Bootstrap starting.
I20260811 11:50:17.630436 7877 tablet_bootstrap.cc:492] T 2b4d807ae5bc44ef84eb5b451463da51 P de1c7edbf94e40bb8e509194b138cf56: Bootstrap starting.
I20260811 11:50:17.630975 7875 tablet_bootstrap.cc:654] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:17.631306 7877 tablet_bootstrap.cc:654] T 2b4d807ae5bc44ef84eb5b451463da51 P de1c7edbf94e40bb8e509194b138cf56: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:17.631884 7878 tablet_bootstrap.cc:492] T 2b4d807ae5bc44ef84eb5b451463da51 P eaf3929d5afa44cdb1aac4e306a3d835: Bootstrap starting.
I20260811 11:50:17.632267 7875 tablet_bootstrap.cc:492] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad: No bootstrap required, opened a new log
I20260811 11:50:17.632369 7875 ts_tablet_manager.cc:1397] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 11:50:17.632377 7877 tablet_bootstrap.cc:492] T 2b4d807ae5bc44ef84eb5b451463da51 P de1c7edbf94e40bb8e509194b138cf56: No bootstrap required, opened a new log
I20260811 11:50:17.632462 7877 ts_tablet_manager.cc:1397] T 2b4d807ae5bc44ef84eb5b451463da51 P de1c7edbf94e40bb8e509194b138cf56: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:50:17.632759 7878 tablet_bootstrap.cc:654] T 2b4d807ae5bc44ef84eb5b451463da51 P eaf3929d5afa44cdb1aac4e306a3d835: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:17.632838 7875 raft_consensus.cc:348] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:17.632902 7877 raft_consensus.cc:348] T 2b4d807ae5bc44ef84eb5b451463da51 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:17.632974 7875 raft_consensus.cc:374] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:17.633010 7875 raft_consensus.cc:729] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4edf677c3f24f6a9b0d7054252810ad, State: Initialized, Role: FOLLOWER
I20260811 11:50:17.633013 7877 raft_consensus.cc:374] T 2b4d807ae5bc44ef84eb5b451463da51 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:17.633047 7877 raft_consensus.cc:729] T 2b4d807ae5bc44ef84eb5b451463da51 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de1c7edbf94e40bb8e509194b138cf56, State: Initialized, Role: FOLLOWER
I20260811 11:50:17.633167 7877 consensus_queue.cc:260] T 2b4d807ae5bc44ef84eb5b451463da51 P de1c7edbf94e40bb8e509194b138cf56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:17.633208 7875 consensus_queue.cc:260] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:17.633551 7877 ts_tablet_manager.cc:1428] T 2b4d807ae5bc44ef84eb5b451463da51 P de1c7edbf94e40bb8e509194b138cf56: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:17.633646 7875 ts_tablet_manager.cc:1428] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260811 11:50:17.633963 7878 tablet_bootstrap.cc:492] T 2b4d807ae5bc44ef84eb5b451463da51 P eaf3929d5afa44cdb1aac4e306a3d835: No bootstrap required, opened a new log
I20260811 11:50:17.634055 7878 ts_tablet_manager.cc:1397] T 2b4d807ae5bc44ef84eb5b451463da51 P eaf3929d5afa44cdb1aac4e306a3d835: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260811 11:50:17.634454 7878 raft_consensus.cc:348] T 2b4d807ae5bc44ef84eb5b451463da51 P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:17.634569 7878 raft_consensus.cc:374] T 2b4d807ae5bc44ef84eb5b451463da51 P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:17.634611 7878 raft_consensus.cc:729] T 2b4d807ae5bc44ef84eb5b451463da51 P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf3929d5afa44cdb1aac4e306a3d835, State: Initialized, Role: FOLLOWER
I20260811 11:50:17.634742 7878 consensus_queue.cc:260] T 2b4d807ae5bc44ef84eb5b451463da51 P eaf3929d5afa44cdb1aac4e306a3d835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:17.635005 7878 ts_tablet_manager.cc:1428] T 2b4d807ae5bc44ef84eb5b451463da51 P eaf3929d5afa44cdb1aac4e306a3d835: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 11:50:17.755764 7885 raft_consensus.cc:482] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:50:17.755923 7885 raft_consensus.cc:504] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:17.757064 7885 leader_election.cc:288] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de1c7edbf94e40bb8e509194b138cf56 (127.3.221.65:41349), eaf3929d5afa44cdb1aac4e306a3d835 (127.3.221.66:45007)
I20260811 11:50:17.761277 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:17.757464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:45007 (local address 127.3.221.67:46341)
0811 11:50:17.757869 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:17.757882 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:17.757998 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:17.758232 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:17.758234 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:17.758279 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:17.758451 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:17.758459 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:17.759224 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:17.759228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:17.760129 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:17.760140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:17.760796 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:17.760800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:17.760905 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:17.760927 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 11:50:17.760996 (+ 69us) client_negotiation.cc:240] Negotiation successful
0811 11:50:17.761072 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":29,"spinlock_wait_cycles":13696,"thread_start_us":116,"threads_started":1}
I20260811 11:50:17.761267 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:17.757488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:46341 (local address 127.3.221.66:45007)
0811 11:50:17.757810 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:17.757819 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:17.757972 (+ 153us) server_negotiation.cc:407] Connection header received
0811 11:50:17.758087 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:17.758090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:17.758133 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:17.758206 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:17.758572 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:17.759103 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:17.760266 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:17.760675 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:17.760787 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:17.760847 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:17.761032 (+ 185us) server_negotiation.cc:299] Negotiation successful
0811 11:50:17.761105 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":131,"threads_started":1}
I20260811 11:50:17.761329 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:17.757476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:52999 (local address 127.3.221.65:41349)
0811 11:50:17.757809 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:17.757818 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:17.757924 (+ 106us) server_negotiation.cc:407] Connection header received
0811 11:50:17.758071 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:17.758074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:17.758117 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:17.758199 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:17.758586 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:17.759231 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:17.760347 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:17.760745 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:17.760858 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:17.760909 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:17.761090 (+ 181us) server_negotiation.cc:299] Negotiation successful
0811 11:50:17.761165 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":113,"threads_started":1}
I20260811 11:50:17.761575 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:17.757390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:41349 (local address 127.3.221.67:52999)
0811 11:50:17.757808 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:17.757823 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:17.757963 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:17.758229 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:17.758232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:17.758277 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:17.758423 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:17.758431 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:17.759372 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:17.759376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:17.760217 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:17.760228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:17.760868 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:17.760872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:17.760973 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:17.760995 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 11:50:17.761061 (+ 66us) client_negotiation.cc:240] Negotiation successful
0811 11:50:17.761131 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"spinlock_wait_cycles":35840,"thread_start_us":147,"threads_started":1}
I20260811 11:50:17.761915 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4d807ae5bc44ef84eb5b451463da51" candidate_uuid: "c4edf677c3f24f6a9b0d7054252810ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" is_pre_election: true
I20260811 11:50:17.762039 7133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4d807ae5bc44ef84eb5b451463da51" candidate_uuid: "c4edf677c3f24f6a9b0d7054252810ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de1c7edbf94e40bb8e509194b138cf56" is_pre_election: true
I20260811 11:50:17.762092 7382 raft_consensus.cc:2454] T 2b4d807ae5bc44ef84eb5b451463da51 P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4edf677c3f24f6a9b0d7054252810ad in term 0.
I20260811 11:50:17.762171 7133 raft_consensus.cc:2454] T 2b4d807ae5bc44ef84eb5b451463da51 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4edf677c3f24f6a9b0d7054252810ad in term 0.
I20260811 11:50:17.762535 7505 leader_election.cc:302] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c4edf677c3f24f6a9b0d7054252810ad, de1c7edbf94e40bb8e509194b138cf56, eaf3929d5afa44cdb1aac4e306a3d835; no voters:
I20260811 11:50:17.762782 7885 raft_consensus.cc:2781] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:50:17.762841 7885 raft_consensus.cc:482] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:50:17.762883 7885 raft_consensus.cc:3037] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:17.763909 7885 raft_consensus.cc:504] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:17.764266 7885 leader_election.cc:288] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad [CANDIDATE]: Term 1 election: Requested vote from peers de1c7edbf94e40bb8e509194b138cf56 (127.3.221.65:41349), eaf3929d5afa44cdb1aac4e306a3d835 (127.3.221.66:45007)
I20260811 11:50:17.764648 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4d807ae5bc44ef84eb5b451463da51" candidate_uuid: "c4edf677c3f24f6a9b0d7054252810ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf3929d5afa44cdb1aac4e306a3d835"
I20260811 11:50:17.764648 7133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4d807ae5bc44ef84eb5b451463da51" candidate_uuid: "c4edf677c3f24f6a9b0d7054252810ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de1c7edbf94e40bb8e509194b138cf56"
I20260811 11:50:17.764779 7382 raft_consensus.cc:3037] T 2b4d807ae5bc44ef84eb5b451463da51 P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:17.764801 7133 raft_consensus.cc:3037] T 2b4d807ae5bc44ef84eb5b451463da51 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:17.765959 7382 raft_consensus.cc:2454] T 2b4d807ae5bc44ef84eb5b451463da51 P eaf3929d5afa44cdb1aac4e306a3d835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4edf677c3f24f6a9b0d7054252810ad in term 1.
I20260811 11:50:17.765959 7133 raft_consensus.cc:2454] T 2b4d807ae5bc44ef84eb5b451463da51 P de1c7edbf94e40bb8e509194b138cf56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4edf677c3f24f6a9b0d7054252810ad in term 1.
I20260811 11:50:17.766260 7505 leader_election.cc:302] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4edf677c3f24f6a9b0d7054252810ad, de1c7edbf94e40bb8e509194b138cf56; no voters:
I20260811 11:50:17.766449 7885 raft_consensus.cc:2781] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:17.766686 7885 raft_consensus.cc:686] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad [term 1 LEADER]: Becoming Leader. State: Replica: c4edf677c3f24f6a9b0d7054252810ad, State: Running, Role: LEADER
I20260811 11:50:17.766954 7885 consensus_queue.cc:237] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:17.769490 6865 catalog_manager.cc:5495] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad reported cstate change: term changed from 0 to 1, leader changed from <none> to c4edf677c3f24f6a9b0d7054252810ad (127.3.221.67). New cstate: current_term: 1 leader_uuid: "c4edf677c3f24f6a9b0d7054252810ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } health_report { overall_health: HEALTHY } } }
I20260811 11:50:17.800602 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:17.796486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45966 (local address 127.3.221.67:43685)
0811 11:50:17.796804 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:17.796812 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:17.796827 (+ 15us) server_negotiation.cc:407] Connection header received
0811 11:50:17.796927 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:17.796931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:17.796996 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:17.797145 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:17.797403 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:17.798241 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:17.799083 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:17.799406 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:17.799506 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:17.799738 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:17.800226 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:17.800314 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:17.800375 (+ 61us) server_negotiation.cc:299] Negotiation successful
0811 11:50:17.800429 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":141,"threads_started":1}
I20260811 11:50:17.811678 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:17.807895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:45007 (local address 127.3.221.67:46847)
0811 11:50:17.808246 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:17.808259 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:17.808383 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:17.808598 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:17.808601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:17.808607 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:50:17.808725 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:17.808730 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:17.809495 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:17.809498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:17.810353 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:17.810367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:17.810800 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:17.810804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:17.810926 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:17.810981 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:50:17.811334 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:50:17.811344 (+ 10us) client_negotiation.cc:769] Sending connection context
0811 11:50:17.811453 (+ 109us) client_negotiation.cc:240] Negotiation successful
0811 11:50:17.811534 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":110,"threads_started":1}
I20260811 11:50:17.811666 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:17.807996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:46847 (local address 127.3.221.66:45007)
0811 11:50:17.808152 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:17.808157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:17.808363 (+ 206us) server_negotiation.cc:407] Connection header received
0811 11:50:17.808467 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:17.808469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:17.808517 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:17.808579 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:17.808825 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:17.809379 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:17.810468 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:17.810690 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:17.810781 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:17.811119 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:17.811225 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:17.811297 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:17.811454 (+ 157us) server_negotiation.cc:299] Negotiation successful
0811 11:50:17.811518 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 11:50:17.818179 7133 raft_consensus.cc:1261] T 2b4d807ae5bc44ef84eb5b451463da51 P de1c7edbf94e40bb8e509194b138cf56 [term 1 FOLLOWER]: Refusing update from remote peer c4edf677c3f24f6a9b0d7054252810ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:17.818290 7382 raft_consensus.cc:1261] T 2b4d807ae5bc44ef84eb5b451463da51 P eaf3929d5afa44cdb1aac4e306a3d835 [term 1 FOLLOWER]: Refusing update from remote peer c4edf677c3f24f6a9b0d7054252810ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:17.818643 7892 consensus_queue.cc:1035] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:17.818884 7885 consensus_queue.cc:1035] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:17.837001 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:17.833569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:43685 (local address 127.3.221.66:47747)
0811 11:50:17.833844 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:17.833857 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:17.833950 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:17.834204 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:17.834207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:17.834217 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:50:17.834350 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:17.834359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:17.835093 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:17.835096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:17.835707 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:17.835715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:17.836229 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:17.836232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:17.836326 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:17.836363 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:50:17.836690 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:50:17.836698 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 11:50:17.836788 (+ 90us) client_negotiation.cc:240] Negotiation successful
0811 11:50:17.836877 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":118,"threads_started":1}
I20260811 11:50:17.837005 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:17.833627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:47747 (local address 127.3.221.67:43685)
0811 11:50:17.833744 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:17.833748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:17.833937 (+ 189us) server_negotiation.cc:407] Connection header received
0811 11:50:17.834058 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:17.834061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:17.834106 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:17.834174 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:17.834475 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:17.834984 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:17.835826 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:17.836053 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:17.836145 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:17.836485 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:17.836573 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:17.836647 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:17.836802 (+ 155us) server_negotiation.cc:299] Negotiation successful
0811 11:50:17.836878 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260811 11:50:17.862279 6865 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:17.862453 6865 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:17.864706 6865 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a: Sending DeleteTablet for 3 replicas of tablet 2b4d807ae5bc44ef84eb5b451463da51
I20260811 11:50:17.865379 7098 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b4d807ae5bc44ef84eb5b451463da51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:17 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:39378
I20260811 11:50:17.865479 7353 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b4d807ae5bc44ef84eb5b451463da51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:17 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:54104
I20260811 11:50:17.865607 7856 tablet_replica.cc:330] T 2b4d807ae5bc44ef84eb5b451463da51 P de1c7edbf94e40bb8e509194b138cf56: stopping tablet replica
I20260811 11:50:17.865711 7856 raft_consensus.cc:2229] T 2b4d807ae5bc44ef84eb5b451463da51 P de1c7edbf94e40bb8e509194b138cf56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:17.865833 7856 raft_consensus.cc:2258] T 2b4d807ae5bc44ef84eb5b451463da51 P de1c7edbf94e40bb8e509194b138cf56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:17.865849 7854 tablet_replica.cc:330] T 2b4d807ae5bc44ef84eb5b451463da51 P eaf3929d5afa44cdb1aac4e306a3d835: stopping tablet replica
I20260811 11:50:17.865873 7580 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b4d807ae5bc44ef84eb5b451463da51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:17 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:49786
I20260811 11:50:17.865990 7854 raft_consensus.cc:2229] T 2b4d807ae5bc44ef84eb5b451463da51 P eaf3929d5afa44cdb1aac4e306a3d835 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:17.866057 7857 tablet_replica.cc:330] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad: stopping tablet replica
I20260811 11:50:17.866110 7854 raft_consensus.cc:2258] T 2b4d807ae5bc44ef84eb5b451463da51 P eaf3929d5afa44cdb1aac4e306a3d835 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:17.866139 7857 raft_consensus.cc:2229] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:50:17.866465 7857 raft_consensus.cc:2258] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:17.866951 7854 ts_tablet_manager.cc:1905] T 2b4d807ae5bc44ef84eb5b451463da51 P eaf3929d5afa44cdb1aac4e306a3d835: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:17.867141 7857 ts_tablet_manager.cc:1905] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:17.867987 6865 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:50:17.868462 6865 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:50:17.869747 7854 ts_tablet_manager.cc:1918] T 2b4d807ae5bc44ef84eb5b451463da51 P eaf3929d5afa44cdb1aac4e306a3d835: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:50:17.869743 7857 ts_tablet_manager.cc:1918] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:50:17.869830 7857 log.cc:1196] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/2b4d807ae5bc44ef84eb5b451463da51
I20260811 11:50:17.869830 7854 log.cc:1196] T 2b4d807ae5bc44ef84eb5b451463da51 P eaf3929d5afa44cdb1aac4e306a3d835: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/2b4d807ae5bc44ef84eb5b451463da51
I20260811 11:50:17.870079 7854 ts_tablet_manager.cc:1939] T 2b4d807ae5bc44ef84eb5b451463da51 P eaf3929d5afa44cdb1aac4e306a3d835: Deleting consensus metadata
I20260811 11:50:17.870079 7857 ts_tablet_manager.cc:1939] T 2b4d807ae5bc44ef84eb5b451463da51 P c4edf677c3f24f6a9b0d7054252810ad: Deleting consensus metadata
I20260811 11:50:17.870810 6845 catalog_manager.cc:4841] TS eaf3929d5afa44cdb1aac4e306a3d835 (127.3.221.66:45007): tablet 2b4d807ae5bc44ef84eb5b451463da51 (table test-table [id=0a6607882bec4b9fa0e666817d596cd8]) successfully deleted
I20260811 11:50:17.871428 6844 catalog_manager.cc:4841] TS c4edf677c3f24f6a9b0d7054252810ad (127.3.221.67:43685): tablet 2b4d807ae5bc44ef84eb5b451463da51 (table test-table [id=0a6607882bec4b9fa0e666817d596cd8]) successfully deleted
I20260811 11:50:17.872145 7856 ts_tablet_manager.cc:1905] T 2b4d807ae5bc44ef84eb5b451463da51 P de1c7edbf94e40bb8e509194b138cf56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:17.874943 7856 ts_tablet_manager.cc:1918] T 2b4d807ae5bc44ef84eb5b451463da51 P de1c7edbf94e40bb8e509194b138cf56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:50:17.875038 7856 log.cc:1196] T 2b4d807ae5bc44ef84eb5b451463da51 P de1c7edbf94e40bb8e509194b138cf56: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-0/wal/wals/2b4d807ae5bc44ef84eb5b451463da51
I20260811 11:50:17.874949 7098 tablet_service.cc:1467] Processing CreateTablet for tablet 602e56cfc68246ce8372d28590820465 (DEFAULT_TABLE table=test-table [id=e2227ba93de34b4a98624f84defc0360]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:17.875192 7353 tablet_service.cc:1467] Processing CreateTablet for tablet 602e56cfc68246ce8372d28590820465 (DEFAULT_TABLE table=test-table [id=e2227ba93de34b4a98624f84defc0360]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:17.875299 7856 ts_tablet_manager.cc:1939] T 2b4d807ae5bc44ef84eb5b451463da51 P de1c7edbf94e40bb8e509194b138cf56: Deleting consensus metadata
I20260811 11:50:17.875414 7098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 602e56cfc68246ce8372d28590820465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:17.875455 7353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 602e56cfc68246ce8372d28590820465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:17.877607 7878 tablet_bootstrap.cc:492] T 602e56cfc68246ce8372d28590820465 P eaf3929d5afa44cdb1aac4e306a3d835: Bootstrap starting.
I20260811 11:50:17.877626 7877 tablet_bootstrap.cc:492] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56: Bootstrap starting.
I20260811 11:50:17.878350 7878 tablet_bootstrap.cc:654] T 602e56cfc68246ce8372d28590820465 P eaf3929d5afa44cdb1aac4e306a3d835: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:17.878486 7877 tablet_bootstrap.cc:654] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:17.879202 6840 catalog_manager.cc:4841] TS de1c7edbf94e40bb8e509194b138cf56 (127.3.221.65:41349): tablet 2b4d807ae5bc44ef84eb5b451463da51 (table test-table [id=0a6607882bec4b9fa0e666817d596cd8]) successfully deleted
I20260811 11:50:17.879421 7580 tablet_service.cc:1467] Processing CreateTablet for tablet 602e56cfc68246ce8372d28590820465 (DEFAULT_TABLE table=test-table [id=e2227ba93de34b4a98624f84defc0360]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:17.879514 7878 tablet_bootstrap.cc:492] T 602e56cfc68246ce8372d28590820465 P eaf3929d5afa44cdb1aac4e306a3d835: No bootstrap required, opened a new log
I20260811 11:50:17.879585 7878 ts_tablet_manager.cc:1397] T 602e56cfc68246ce8372d28590820465 P eaf3929d5afa44cdb1aac4e306a3d835: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260811 11:50:17.879700 7580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 602e56cfc68246ce8372d28590820465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:17.880074 7878 raft_consensus.cc:348] T 602e56cfc68246ce8372d28590820465 P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:17.880182 7878 raft_consensus.cc:374] T 602e56cfc68246ce8372d28590820465 P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:17.880208 7878 raft_consensus.cc:729] T 602e56cfc68246ce8372d28590820465 P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf3929d5afa44cdb1aac4e306a3d835, State: Initialized, Role: FOLLOWER
I20260811 11:50:17.880312 7878 consensus_queue.cc:260] T 602e56cfc68246ce8372d28590820465 P eaf3929d5afa44cdb1aac4e306a3d835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:17.880549 7878 ts_tablet_manager.cc:1428] T 602e56cfc68246ce8372d28590820465 P eaf3929d5afa44cdb1aac4e306a3d835: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 11:50:17.880579 7877 tablet_bootstrap.cc:492] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56: No bootstrap required, opened a new log
I20260811 11:50:17.880676 7877 ts_tablet_manager.cc:1397] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 11:50:17.881045 7877 raft_consensus.cc:348] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:17.881155 7877 raft_consensus.cc:374] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:17.881183 7877 raft_consensus.cc:729] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de1c7edbf94e40bb8e509194b138cf56, State: Initialized, Role: FOLLOWER
I20260811 11:50:17.881332 7877 consensus_queue.cc:260] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:17.881595 7877 ts_tablet_manager.cc:1428] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:17.882169 7875 tablet_bootstrap.cc:492] T 602e56cfc68246ce8372d28590820465 P c4edf677c3f24f6a9b0d7054252810ad: Bootstrap starting.
I20260811 11:50:17.883150 7875 tablet_bootstrap.cc:654] T 602e56cfc68246ce8372d28590820465 P c4edf677c3f24f6a9b0d7054252810ad: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:17.884459 7875 tablet_bootstrap.cc:492] T 602e56cfc68246ce8372d28590820465 P c4edf677c3f24f6a9b0d7054252810ad: No bootstrap required, opened a new log
I20260811 11:50:17.884554 7875 ts_tablet_manager.cc:1397] T 602e56cfc68246ce8372d28590820465 P c4edf677c3f24f6a9b0d7054252810ad: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260811 11:50:17.885000 7875 raft_consensus.cc:348] T 602e56cfc68246ce8372d28590820465 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:17.885125 7875 raft_consensus.cc:374] T 602e56cfc68246ce8372d28590820465 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:17.885150 7875 raft_consensus.cc:729] T 602e56cfc68246ce8372d28590820465 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4edf677c3f24f6a9b0d7054252810ad, State: Initialized, Role: FOLLOWER
I20260811 11:50:17.885279 7875 consensus_queue.cc:260] T 602e56cfc68246ce8372d28590820465 P c4edf677c3f24f6a9b0d7054252810ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:17.885528 7875 ts_tablet_manager.cc:1428] T 602e56cfc68246ce8372d28590820465 P c4edf677c3f24f6a9b0d7054252810ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 11:50:17.989391 7843 raft_consensus.cc:482] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:50:17.989559 7843 raft_consensus.cc:504] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:17.990031 7843 leader_election.cc:288] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf3929d5afa44cdb1aac4e306a3d835 (127.3.221.66:45007), c4edf677c3f24f6a9b0d7054252810ad (127.3.221.67:43685)
I20260811 11:50:17.990525 7606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "602e56cfc68246ce8372d28590820465" candidate_uuid: "de1c7edbf94e40bb8e509194b138cf56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4edf677c3f24f6a9b0d7054252810ad" is_pre_election: true
I20260811 11:50:17.990571 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "602e56cfc68246ce8372d28590820465" candidate_uuid: "de1c7edbf94e40bb8e509194b138cf56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" is_pre_election: true
I20260811 11:50:17.990693 7606 raft_consensus.cc:2454] T 602e56cfc68246ce8372d28590820465 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de1c7edbf94e40bb8e509194b138cf56 in term 0.
I20260811 11:50:17.990757 7382 raft_consensus.cc:2454] T 602e56cfc68246ce8372d28590820465 P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de1c7edbf94e40bb8e509194b138cf56 in term 0.
I20260811 11:50:17.991024 7012 leader_election.cc:302] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4edf677c3f24f6a9b0d7054252810ad, de1c7edbf94e40bb8e509194b138cf56; no voters:
I20260811 11:50:17.991210 7843 raft_consensus.cc:2781] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:50:17.991285 7843 raft_consensus.cc:482] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:50:17.991330 7843 raft_consensus.cc:3037] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:17.992404 7843 raft_consensus.cc:504] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:17.992765 7843 leader_election.cc:288] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56 [CANDIDATE]: Term 1 election: Requested vote from peers eaf3929d5afa44cdb1aac4e306a3d835 (127.3.221.66:45007), c4edf677c3f24f6a9b0d7054252810ad (127.3.221.67:43685)
I20260811 11:50:17.993263 7606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "602e56cfc68246ce8372d28590820465" candidate_uuid: "de1c7edbf94e40bb8e509194b138cf56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4edf677c3f24f6a9b0d7054252810ad"
I20260811 11:50:17.993260 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "602e56cfc68246ce8372d28590820465" candidate_uuid: "de1c7edbf94e40bb8e509194b138cf56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf3929d5afa44cdb1aac4e306a3d835"
I20260811 11:50:17.993397 7606 raft_consensus.cc:3037] T 602e56cfc68246ce8372d28590820465 P c4edf677c3f24f6a9b0d7054252810ad [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:17.993407 7382 raft_consensus.cc:3037] T 602e56cfc68246ce8372d28590820465 P eaf3929d5afa44cdb1aac4e306a3d835 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:17.994361 7606 raft_consensus.cc:2454] T 602e56cfc68246ce8372d28590820465 P c4edf677c3f24f6a9b0d7054252810ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de1c7edbf94e40bb8e509194b138cf56 in term 1.
I20260811 11:50:17.994660 7012 leader_election.cc:302] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4edf677c3f24f6a9b0d7054252810ad, de1c7edbf94e40bb8e509194b138cf56; no voters:
I20260811 11:50:17.994721 7382 raft_consensus.cc:2454] T 602e56cfc68246ce8372d28590820465 P eaf3929d5afa44cdb1aac4e306a3d835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de1c7edbf94e40bb8e509194b138cf56 in term 1.
I20260811 11:50:17.994853 7843 raft_consensus.cc:2781] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:17.994941 7843 raft_consensus.cc:686] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56 [term 1 LEADER]: Becoming Leader. State: Replica: de1c7edbf94e40bb8e509194b138cf56, State: Running, Role: LEADER
I20260811 11:50:17.995087 7843 consensus_queue.cc:237] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } }
I20260811 11:50:17.996932 6865 catalog_manager.cc:5495] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56 reported cstate change: term changed from 0 to 1, leader changed from <none> to de1c7edbf94e40bb8e509194b138cf56 (127.3.221.65). New cstate: current_term: 1 leader_uuid: "de1c7edbf94e40bb8e509194b138cf56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1c7edbf94e40bb8e509194b138cf56" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 } health_report { overall_health: UNKNOWN } } }
I20260811 11:50:18.019094 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:18.015686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44980 (local address 127.3.221.65:41349)
0811 11:50:18.015800 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:18.015804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:18.015817 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:50:18.015924 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:18.015926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:18.015970 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:18.016028 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:18.016249 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:18.016736 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:18.017935 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:18.018212 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:18.018317 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:18.018628 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:18.018717 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:18.018807 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:18.018894 (+ 87us) server_negotiation.cc:299] Negotiation successful
0811 11:50:18.018950 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260811 11:50:18.026019 7382 raft_consensus.cc:1261] T 602e56cfc68246ce8372d28590820465 P eaf3929d5afa44cdb1aac4e306a3d835 [term 1 FOLLOWER]: Refusing update from remote peer de1c7edbf94e40bb8e509194b138cf56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:18.026011 7606 raft_consensus.cc:1261] T 602e56cfc68246ce8372d28590820465 P c4edf677c3f24f6a9b0d7054252810ad [term 1 FOLLOWER]: Refusing update from remote peer de1c7edbf94e40bb8e509194b138cf56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:18.026424 7735 consensus_queue.cc:1035] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf3929d5afa44cdb1aac4e306a3d835" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 45007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:18.026686 7735 consensus_queue.cc:1035] T 602e56cfc68246ce8372d28590820465 P de1c7edbf94e40bb8e509194b138cf56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4edf677c3f24f6a9b0d7054252810ad" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 11:50:18.041357 7830 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260811 11:50:19.594920 6866 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:19.595086 6866 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:19.597697 6866 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 138578b03cf54dfca455d1456d96ac0a: Sending DeleteTablet for 3 replicas of tablet 602e56cfc68246ce8372d28590820465
I20260811 11:50:19.598346 7098 tablet_service.cc:1514] Processing DeleteTablet for tablet 602e56cfc68246ce8372d28590820465 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:19 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:39378
I20260811 11:50:19.598492 7353 tablet_service.cc:1514] Processing DeleteTablet for tablet 602e56cfc68246ce8372d28590820465 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:19 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:54104
I20260811 11:50:19.598500 7580 tablet_service.cc:1514] Processing DeleteTablet for tablet 602e56cfc68246ce8372d28590820465 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:19 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:49786
I20260811 11:50:19.598853 7958 tablet_replica.cc:330] T 602e56cfc68246ce8372d28590820465 P eaf3929d5afa44cdb1aac4e306a3d835: stopping tablet replica
I20260811 11:50:19.598862 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 6983
I20260811 11:50:19.598991 7958 raft_consensus.cc:2229] T 602e56cfc68246ce8372d28590820465 P eaf3929d5afa44cdb1aac4e306a3d835 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:19.599112 7958 raft_consensus.cc:2258] T 602e56cfc68246ce8372d28590820465 P eaf3929d5afa44cdb1aac4e306a3d835 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:19.599354 7959 tablet_replica.cc:330] T 602e56cfc68246ce8372d28590820465 P c4edf677c3f24f6a9b0d7054252810ad: stopping tablet replica
I20260811 11:50:19.599481 7959 raft_consensus.cc:2229] T 602e56cfc68246ce8372d28590820465 P c4edf677c3f24f6a9b0d7054252810ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:19.599591 7959 raft_consensus.cc:2258] T 602e56cfc68246ce8372d28590820465 P c4edf677c3f24f6a9b0d7054252810ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:19.601657 7959 ts_tablet_manager.cc:1905] T 602e56cfc68246ce8372d28590820465 P c4edf677c3f24f6a9b0d7054252810ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:19.604897 7959 ts_tablet_manager.cc:1918] T 602e56cfc68246ce8372d28590820465 P c4edf677c3f24f6a9b0d7054252810ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 11:50:19.604981 7959 log.cc:1196] T 602e56cfc68246ce8372d28590820465 P c4edf677c3f24f6a9b0d7054252810ad: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/602e56cfc68246ce8372d28590820465
I20260811 11:50:19.605197 7959 ts_tablet_manager.cc:1939] T 602e56cfc68246ce8372d28590820465 P c4edf677c3f24f6a9b0d7054252810ad: Deleting consensus metadata
I20260811 11:50:19.605512 7958 ts_tablet_manager.cc:1905] T 602e56cfc68246ce8372d28590820465 P eaf3929d5afa44cdb1aac4e306a3d835: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:19.605908 6844 catalog_manager.cc:4841] TS c4edf677c3f24f6a9b0d7054252810ad (127.3.221.67:43685): tablet 602e56cfc68246ce8372d28590820465 (table test-table [id=e2227ba93de34b4a98624f84defc0360]) successfully deleted
I20260811 11:50:19.608367 7958 ts_tablet_manager.cc:1918] T 602e56cfc68246ce8372d28590820465 P eaf3929d5afa44cdb1aac4e306a3d835: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 11:50:19.608481 7958 log.cc:1196] T 602e56cfc68246ce8372d28590820465 P eaf3929d5afa44cdb1aac4e306a3d835: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/602e56cfc68246ce8372d28590820465
I20260811 11:50:19.608815 7958 ts_tablet_manager.cc:1939] T 602e56cfc68246ce8372d28590820465 P eaf3929d5afa44cdb1aac4e306a3d835: Deleting consensus metadata
I20260811 11:50:19.609753 6845 catalog_manager.cc:4841] TS eaf3929d5afa44cdb1aac4e306a3d835 (127.3.221.66:45007): tablet 602e56cfc68246ce8372d28590820465 (table test-table [id=e2227ba93de34b4a98624f84defc0360]) successfully deleted
W20260811 11:50:19.622458 6840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.221.65:41349 (error 108)
I20260811 11:50:19.622668 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 7237
I20260811 11:50:19.633971 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:19.631705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:41349 (local address 127.0.0.1:44984)
0811 11:50:19.633767 (+ 2062us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:19.633864 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.65:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1970,"thread_start_us":1856,"threads_started":1}
W20260811 11:50:19.634256 6840 catalog_manager.cc:4584] TS de1c7edbf94e40bb8e509194b138cf56 (127.3.221.65:41349): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 602e56cfc68246ce8372d28590820465: Network error: Client connection negotiation failed: client connection to 127.3.221.65:41349: connect: Connection refused (error 111)
I20260811 11:50:19.641501 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 7469
I20260811 11:50:19.656773 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 6808
2026-08-11T11:50:19Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6666 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:50:19 dist-test-slave-bndk krb5kdc[7971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:50:19 dist-test-slave-bndk krb5kdc[7971](info): set up 2 sockets
Aug 11 11:50:19 dist-test-slave-bndk krb5kdc[7971](info): commencing operation
krb5kdc: starting...
W20260811 11:50:20.850594 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:50:20 dist-test-slave-bndk krb5kdc[7971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:50:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:50:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:50:20.984288 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:44933
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:44933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:21.101312 7995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:21.101608 7995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:21.101663 7995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:21.105682 7995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:50:21.105759 7995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:21.105785 7995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:21.105808 7995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:50:21.105827 7995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:50:21.112471 7995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:44933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:44933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:21.113790 7995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:21.114940 7995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:21.124568 8006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:21.124511 8003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:21.124552 7995 server_base.cc:1034] running on GCE node
W20260811 11:50:21.124511 8010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:21.125476 7995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:21.126658 7995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:21.127830 7995 hybrid_clock.cc:648] HybridClock initialized: now 1786449021127813 us; error 32 us; skew 500 ppm
Aug 11 11:50:21 dist-test-slave-bndk krb5kdc[7971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449021, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:21.130821 7995 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:50:21.131831 7995 webserver.cc:460] Webserver started at http://127.3.221.126:44043/ using document root <none> and password file <none>
I20260811 11:50:21.132390 7995 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:21.132450 7995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:21.132663 7995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:21.134893 7995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "971f872b3e104591a90d9534083353ec"
format_stamp: "Formatted at 2026-08-11 11:50:21 on dist-test-slave-bndk"
server_key: "db430fcd77f08a6dc78e001bdbaea9f8"
server_key_iv: "403d385fb64a78972cd01755064d9d24"
server_key_version: "encryptionkey@0"
I20260811 11:50:21.135478 7995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "971f872b3e104591a90d9534083353ec"
format_stamp: "Formatted at 2026-08-11 11:50:21 on dist-test-slave-bndk"
server_key: "db430fcd77f08a6dc78e001bdbaea9f8"
server_key_iv: "403d385fb64a78972cd01755064d9d24"
server_key_version: "encryptionkey@0"
I20260811 11:50:21.139451 7995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260811 11:50:21.142042 8018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:21.143312 7995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 11:50:21.143438 7995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "971f872b3e104591a90d9534083353ec"
format_stamp: "Formatted at 2026-08-11 11:50:21 on dist-test-slave-bndk"
server_key: "db430fcd77f08a6dc78e001bdbaea9f8"
server_key_iv: "403d385fb64a78972cd01755064d9d24"
server_key_version: "encryptionkey@0"
I20260811 11:50:21.143556 7995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:21.177863 7995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:21.181313 7995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:21.181512 7995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:21.188920 7995 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:44933
I20260811 11:50:21.188990 8102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:44933 every 8 connection(s)
I20260811 11:50:21.189992 7995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:50:21.192256 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 7995
I20260811 11:50:21.192351 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:50:21.192626 3957 external_mini_cluster.cc:1428] Setting key f16925e75ddaa047eda42a31f18483d2
I20260811 11:50:21.193008 8103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:21.199520 8103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec: Bootstrap starting.
Aug 11 11:50:21 dist-test-slave-bndk krb5kdc[7971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:21.202237 8103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:21.203106 8103 log.cc:824] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:21.205145 8103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec: No bootstrap required, opened a new log
I20260811 11:50:21.207054 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:21.194175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59342 (local address 127.3.221.126:44933)
0811 11:50:21.194649 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:21.194665 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:21.194703 (+ 38us) server_negotiation.cc:407] Connection header received
0811 11:50:21.195376 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:21.195393 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:21.195721 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:21.196042 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:21.196896 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:21.197986 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:21.198884 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:21.199365 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:21.199526 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:21.202330 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:21.202335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:21.202353 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:21.202391 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:21.203978 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:21.204418 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:21.204423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:21.204428 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:21.204539 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:21.204816 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:21.204818 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:21.204819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:21.205244 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:21.205415 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:21.205765 (+ 350us) server_negotiation.cc:299] Negotiation successful
0811 11:50:21.206096 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":148,"threads_started":1}
I20260811 11:50:21.208571 8103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971f872b3e104591a90d9534083353ec" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 44933 } }
I20260811 11:50:21.208853 8103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:21.208923 8103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f872b3e104591a90d9534083353ec, State: Initialized, Role: FOLLOWER
I20260811 11:50:21.209388 8103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971f872b3e104591a90d9534083353ec" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 44933 } }
I20260811 11:50:21.209533 8103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:50:21.209581 8103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:50:21.209656 8103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:21.210536 8103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971f872b3e104591a90d9534083353ec" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 44933 } }
I20260811 11:50:21.210842 8103 leader_election.cc:302] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 971f872b3e104591a90d9534083353ec; no voters:
I20260811 11:50:21.211122 8103 leader_election.cc:288] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:50:21.211239 8110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:21.211472 8110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec [term 1 LEADER]: Becoming Leader. State: Replica: 971f872b3e104591a90d9534083353ec, State: Running, Role: LEADER
I20260811 11:50:21.211825 8110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971f872b3e104591a90d9534083353ec" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 44933 } }
I20260811 11:50:21.212352 8103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:50:21.214479 8112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 971f872b3e104591a90d9534083353ec. Latest consensus state: current_term: 1 leader_uuid: "971f872b3e104591a90d9534083353ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971f872b3e104591a90d9534083353ec" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 44933 } } }
I20260811 11:50:21.214795 8112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:21.214397 8111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "971f872b3e104591a90d9534083353ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971f872b3e104591a90d9534083353ec" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 44933 } } }
I20260811 11:50:21.215149 8111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:21.215210 8126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:50:21.217999 8126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:50:21.223672 8126 catalog_manager.cc:1347] Generated new cluster ID: 60d77585bdf7425093a8304ab19df6f7
I20260811 11:50:21.223762 8126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:50:21.249167 8126 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:50:21.250535 8126 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:50:21.261348 8126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 971f872b3e104591a90d9534083353ec: Generated new TSK 0
I20260811 11:50:21.262215 8126 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
I20260811 11:50:21.302831 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:44933
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:21.421808 8141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:21.422119 8141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:21.422184 8141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:21.426369 8141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:21.426450 8141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:21.426534 8141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:50:21.432140 8141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:44933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:21.433388 8141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:21.434492 8141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:21.444461 8155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:21.444438 8159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:21.444470 8154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:21.444617 8141 server_base.cc:1034] running on GCE node
I20260811 11:50:21.445189 8141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:21.445930 8141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:21.447130 8141 hybrid_clock.cc:648] HybridClock initialized: now 1786449021447110 us; error 31 us; skew 500 ppm
Aug 11 11:50:21 dist-test-slave-bndk krb5kdc[7971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449021, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:21.450099 8141 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:50:21.451076 8141 webserver.cc:460] Webserver started at http://127.3.221.65:46749/ using document root <none> and password file <none>
I20260811 11:50:21.451683 8141 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:21.451756 8141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:21.451961 8141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:21.454180 8141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "4b59d53b8e5d4db7ba24a667f6a5993d"
format_stamp: "Formatted at 2026-08-11 11:50:21 on dist-test-slave-bndk"
server_key: "e962f6cdad733eeb9a1299e169cf2729"
server_key_iv: "8ba6e1131df201530e17d2958aeb5d7c"
server_key_version: "encryptionkey@0"
I20260811 11:50:21.454766 8141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "4b59d53b8e5d4db7ba24a667f6a5993d"
format_stamp: "Formatted at 2026-08-11 11:50:21 on dist-test-slave-bndk"
server_key: "e962f6cdad733eeb9a1299e169cf2729"
server_key_iv: "8ba6e1131df201530e17d2958aeb5d7c"
server_key_version: "encryptionkey@0"
I20260811 11:50:21.458853 8141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:50:21.461952 8170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:21.463470 8141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:50:21.463617 8141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "4b59d53b8e5d4db7ba24a667f6a5993d"
format_stamp: "Formatted at 2026-08-11 11:50:21 on dist-test-slave-bndk"
server_key: "e962f6cdad733eeb9a1299e169cf2729"
server_key_iv: "8ba6e1131df201530e17d2958aeb5d7c"
server_key_version: "encryptionkey@0"
I20260811 11:50:21.463744 8141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:21.480078 8141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:21.484017 8141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:21.484241 8141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:21.485016 8141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:21.486187 8141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:21.486255 8141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:21.486325 8141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:21.486356 8141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:21.498570 8141 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:42029
I20260811 11:50:21.498623 8358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:42029 every 8 connection(s)
I20260811 11:50:21.499539 8141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:50:21.501436 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 8141
I20260811 11:50:21.501585 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:50:21.501869 3957 external_mini_cluster.cc:1428] Setting key c348dce7875914c1b038b3cb43e50d03
Aug 11 11:50:21 dist-test-slave-bndk krb5kdc[7971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449021, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.221.66@KRBTEST.COM; defaulting to no policy
I20260811 11:50:21.515749 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:21.501618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:37749 (local address 127.3.221.126:44933)
0811 11:50:21.503152 (+ 1534us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:21.503156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:21.503170 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:50:21.504480 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:21.504484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:21.504540 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:21.504627 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:21.506102 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:21.506661 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:21.508302 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:21.508559 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:21.508650 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:21.511580 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:21.511584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:21.511586 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:21.511613 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:21.513095 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:21.513825 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:21.513828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:21.513829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:21.513921 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:21.514430 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:21.514433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:21.514434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:21.514645 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:21.514731 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:21.515413 (+ 682us) server_negotiation.cc:299] Negotiation successful
0811 11:50:21.515568 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1452}
I20260811 11:50:21.516703 8364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:21.502179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:44933 (local address 127.3.221.65:37749)
0811 11:50:21.502747 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:21.502815 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:21.504228 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:21.504774 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:21.504782 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:21.505179 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:21.505918 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:21.505942 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:21.506817 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:21.506820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:21.508147 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:21.508160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:21.508765 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:21.508771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:21.508986 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:21.509662 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:21.509686 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:21.511361 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:21.513243 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:21.513250 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:21.513273 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:21.513679 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:21.514028 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:21.514031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:21.514033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:21.514299 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:21.514761 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:21.514767 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:21.515054 (+ 287us) client_negotiation.cc:769] Sending connection context
0811 11:50:21.515324 (+ 270us) client_negotiation.cc:240] Negotiation successful
0811 11:50:21.515701 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":148,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.66@KRBTEST.COM" created.
I20260811 11:50:21.518066 8359 heartbeater.cc:347] Connected to a master server at 127.3.221.126:44933
I20260811 11:50:21.518357 8359 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:21.519011 8359 heartbeater.cc:511] Master 127.3.221.126:44933 requested a full tablet report, sending...
I20260811 11:50:21.520849 8048 ts_manager.cc:194] Registered new tserver with Master: 4b59d53b8e5d4db7ba24a667f6a5993d (127.3.221.65:42029)
I20260811 11:50:21.522078 8048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:37749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
I20260811 11:50:21.537142 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:44933
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:21.653753 8370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:21.654044 8370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:21.654109 8370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:21.658149 8370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:21.658215 8370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:21.658303 8370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:50:21.663772 8370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:44933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:21.664925 8370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:21.666098 8370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:21.676335 8380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:21.676378 8370 server_base.cc:1034] running on GCE node
W20260811 11:50:21.676383 8381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:21.676563 8383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:21.677008 8370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:21.677752 8370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:21.678929 8370 hybrid_clock.cc:648] HybridClock initialized: now 1786449021678903 us; error 37 us; skew 500 ppm
Aug 11 11:50:21 dist-test-slave-bndk krb5kdc[7971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449021, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:21.681788 8370 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:50:21.682745 8370 webserver.cc:460] Webserver started at http://127.3.221.66:45295/ using document root <none> and password file <none>
I20260811 11:50:21.683303 8370 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:21.683364 8370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:21.683542 8370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:21.685745 8370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "4452907c6a264317ba09cf932709753d"
format_stamp: "Formatted at 2026-08-11 11:50:21 on dist-test-slave-bndk"
server_key: "d75fb03548c22ba5ce6e7236c2ef99e1"
server_key_iv: "92cd57d29f996d71c60c86e8c59166b5"
server_key_version: "encryptionkey@0"
I20260811 11:50:21.686429 8370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "4452907c6a264317ba09cf932709753d"
format_stamp: "Formatted at 2026-08-11 11:50:21 on dist-test-slave-bndk"
server_key: "d75fb03548c22ba5ce6e7236c2ef99e1"
server_key_iv: "92cd57d29f996d71c60c86e8c59166b5"
server_key_version: "encryptionkey@0"
I20260811 11:50:21.690724 8370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 11:50:21.694571 8393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:21.696341 8370 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:50:21.696478 8370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "4452907c6a264317ba09cf932709753d"
format_stamp: "Formatted at 2026-08-11 11:50:21 on dist-test-slave-bndk"
server_key: "d75fb03548c22ba5ce6e7236c2ef99e1"
server_key_iv: "92cd57d29f996d71c60c86e8c59166b5"
server_key_version: "encryptionkey@0"
I20260811 11:50:21.696630 8370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:21.709581 8370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:21.712459 8370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:21.712615 8370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:21.713205 8370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:21.714098 8370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:21.714144 8370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:21.714188 8370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:21.714217 8370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:21.723232 8370 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:39815
I20260811 11:50:21.723258 8586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:39815 every 8 connection(s)
I20260811 11:50:21.724175 8370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:50:21.725050 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 8370
I20260811 11:50:21.725162 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:50:21.725505 3957 external_mini_cluster.cc:1428] Setting key fd759a1f62e8018fe444581ce8c5b3cb
Aug 11 11:50:21 dist-test-slave-bndk krb5kdc[7971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449021, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.67@KRBTEST.COM" created.
I20260811 11:50:21.739435 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:21.726342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:38223 (local address 127.3.221.126:44933)
0811 11:50:21.726479 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:21.726483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:21.727554 (+ 1071us) server_negotiation.cc:407] Connection header received
0811 11:50:21.729319 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:21.729323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:21.729378 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:21.729462 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:21.730726 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:21.731293 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:21.732472 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:21.732761 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:21.732861 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:21.735494 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:21.735498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:21.735501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:21.735530 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:21.737070 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:21.737701 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:21.737705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:21.737707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:21.737797 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:21.738182 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:21.738185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:21.738187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:21.738397 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:21.738498 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:21.739107 (+ 609us) server_negotiation.cc:299] Negotiation successful
0811 11:50:21.739262 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 11:50:21.740468 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:21.726790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:44933 (local address 127.3.221.66:38223)
0811 11:50:21.727353 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:21.727416 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:21.729024 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:21.729671 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:21.729678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:21.730052 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:21.730566 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:21.730582 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:21.731437 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:21.731440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:21.732337 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:21.732349 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:21.732930 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:21.732935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:21.733101 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:21.733780 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:21.733803 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:21.735310 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:21.737256 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:21.737264 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:21.737285 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:21.737578 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:21.737900 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:21.737903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:21.737905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:21.738075 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:21.738510 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:21.738518 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:21.738805 (+ 287us) client_negotiation.cc:769] Sending connection context
0811 11:50:21.739007 (+ 202us) client_negotiation.cc:240] Negotiation successful
0811 11:50:21.739323 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":155,"threads_started":1}
I20260811 11:50:21.741911 8587 heartbeater.cc:347] Connected to a master server at 127.3.221.126:44933
I20260811 11:50:21.742175 8587 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:21.742730 8587 heartbeater.cc:511] Master 127.3.221.126:44933 requested a full tablet report, sending...
I20260811 11:50:21.743973 8048 ts_manager.cc:194] Registered new tserver with Master: 4452907c6a264317ba09cf932709753d (127.3.221.66:39815)
I20260811 11:50:21.744787 8048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:38223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
I20260811 11:50:21.756569 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:44933
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:21.875520 8594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:21.875808 8594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:21.875855 8594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:21.879998 8594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:21.880085 8594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:21.880147 8594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:50:21.885849 8594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:44933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:21.887004 8594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:21.888096 8594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:21.898083 8600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:21.898078 8605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:21.898123 8599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:21.898262 8594 server_base.cc:1034] running on GCE node
I20260811 11:50:21.898823 8594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:21.899528 8594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:21.900689 8594 hybrid_clock.cc:648] HybridClock initialized: now 1786449021900650 us; error 44 us; skew 500 ppm
Aug 11 11:50:21 dist-test-slave-bndk krb5kdc[7971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449021, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:21.903693 8594 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:50:21.904608 8594 webserver.cc:460] Webserver started at http://127.3.221.67:35775/ using document root <none> and password file <none>
I20260811 11:50:21.905186 8594 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:21.905292 8594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:21.905517 8594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:21.907635 8594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "cb0e6883766442eaa6317b2168ae6f5b"
format_stamp: "Formatted at 2026-08-11 11:50:21 on dist-test-slave-bndk"
server_key: "9ecd4d17d5495e1dcb2c04bc55677c63"
server_key_iv: "5984eeee359d108554660176a43f03f0"
server_key_version: "encryptionkey@0"
I20260811 11:50:21.908221 8594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "cb0e6883766442eaa6317b2168ae6f5b"
format_stamp: "Formatted at 2026-08-11 11:50:21 on dist-test-slave-bndk"
server_key: "9ecd4d17d5495e1dcb2c04bc55677c63"
server_key_iv: "5984eeee359d108554660176a43f03f0"
server_key_version: "encryptionkey@0"
I20260811 11:50:21.912128 8594 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 11:50:21.914691 8618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:21.915958 8594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 11:50:21.916059 8594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "cb0e6883766442eaa6317b2168ae6f5b"
format_stamp: "Formatted at 2026-08-11 11:50:21 on dist-test-slave-bndk"
server_key: "9ecd4d17d5495e1dcb2c04bc55677c63"
server_key_iv: "5984eeee359d108554660176a43f03f0"
server_key_version: "encryptionkey@0"
I20260811 11:50:21.916148 8594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:21.936115 8594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:21.939381 8594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:21.939555 8594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:21.940120 8594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:21.940958 8594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:21.941006 8594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:21.941047 8594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:21.941061 8594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:21.949929 8594 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:35591
I20260811 11:50:21.949954 8806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:35591 every 8 connection(s)
I20260811 11:50:21.950871 8594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:50:21.954744 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 8594
I20260811 11:50:21.954881 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:50:21.955160 3957 external_mini_cluster.cc:1428] Setting key b4e7673dff637437e1062e967f4d5649
Aug 11 11:50:21 dist-test-slave-bndk krb5kdc[7971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449021, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:21.965772 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:21.952803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:34223 (local address 127.3.221.126:44933)
0811 11:50:21.952961 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:21.952966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:21.953925 (+ 959us) server_negotiation.cc:407] Connection header received
0811 11:50:21.955389 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:21.955393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:21.955459 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:21.955555 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:21.956996 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:21.957693 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:21.959034 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:21.959300 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:21.959397 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:21.962101 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:21.962104 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:21.962107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:21.962137 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:21.963479 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:21.964061 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:21.964064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:21.964066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:21.964166 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:21.964528 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:21.964531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:21.964532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:21.964737 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:21.964820 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:21.965514 (+ 694us) server_negotiation.cc:299] Negotiation successful
0811 11:50:21.965642 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 11:50:21.966648 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:21.953254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:44933 (local address 127.3.221.67:34223)
0811 11:50:21.953748 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:21.953796 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:21.955088 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:21.955789 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:21.955798 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:21.956244 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:21.956851 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:21.956868 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:21.957824 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:21.957827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:21.958889 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:21.958902 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:21.959488 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:21.959491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:21.959651 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:21.960300 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:21.960318 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:21.961908 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:21.963616 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:21.963622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:21.963638 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:21.963938 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:21.964285 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:21.964287 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:21.964289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:21.964453 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:21.964869 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:21.964875 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:21.965168 (+ 293us) client_negotiation.cc:769] Sending connection context
0811 11:50:21.965437 (+ 269us) client_negotiation.cc:240] Negotiation successful
0811 11:50:21.965747 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":138,"threads_started":1}
I20260811 11:50:21.967939 8810 heartbeater.cc:347] Connected to a master server at 127.3.221.126:44933
I20260811 11:50:21.968218 8810 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:21.968873 8810 heartbeater.cc:511] Master 127.3.221.126:44933 requested a full tablet report, sending...
I20260811 11:50:21.969991 8048 ts_manager.cc:194] Registered new tserver with Master: cb0e6883766442eaa6317b2168ae6f5b (127.3.221.67:35591)
I20260811 11:50:21.970575 8048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:34223
I20260811 11:50:21.981472 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 11:50:21.997315 8108 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:21.995795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59348 (local address 127.3.221.126:44933)
0811 11:50:21.995966 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:21.995969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:21.996087 (+ 118us) server_negotiation.cc:407] Connection header received
0811 11:50:21.996217 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:21.996219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:21.996266 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:21.996351 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:21.997132 (+ 781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59348: BlockingRecv error: recv got EOF from 127.0.0.1:59348 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 11:50:21.998418 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 8141
I20260811 11:50:22.029736 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 8370
I20260811 11:50:22.042752 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 8594
I20260811 11:50:22.060627 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 7995
2026-08-11T11:50:22Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2405 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:50:22 dist-test-slave-bndk krb5kdc[8834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:50:22 dist-test-slave-bndk krb5kdc[8834](info): set up 2 sockets
Aug 11 11:50:22 dist-test-slave-bndk krb5kdc[8834](info): commencing operation
krb5kdc: starting...
W20260811 11:50:23.235299 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:50:23 dist-test-slave-bndk krb5kdc[8834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:50:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:50:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:50:23.376174 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:37439
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:33129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:37439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:23.492666 8849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:23.492983 8849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:23.493047 8849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:23.497351 8849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:50:23.497437 8849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:23.497464 8849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:23.497488 8849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:50:23.497504 8849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:50:23.503130 8849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:33129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:37439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:37439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:23.504328 8849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:23.505455 8849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:23.514336 8858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:23.514326 8855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:23.514555 8849 server_base.cc:1034] running on GCE node
W20260811 11:50:23.514326 8860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:23.515316 8849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:23.516528 8849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:23.517726 8849 hybrid_clock.cc:648] HybridClock initialized: now 1786449023517684 us; error 55 us; skew 500 ppm
Aug 11 11:50:23 dist-test-slave-bndk krb5kdc[8834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449023, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:23.520756 8849 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:50:23.521806 8849 webserver.cc:460] Webserver started at http://127.3.221.126:33249/ using document root <none> and password file <none>
I20260811 11:50:23.522413 8849 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:23.522491 8849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:23.522706 8849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:23.524925 8849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "570c763f2fd74d8da48b0bec7613fefe"
format_stamp: "Formatted at 2026-08-11 11:50:23 on dist-test-slave-bndk"
server_key: "8dec5a0ac31c6f421429bb9ba3b36bb3"
server_key_iv: "bdf5d0f9beb6739a1af9c2ae285e0341"
server_key_version: "encryptionkey@0"
I20260811 11:50:23.525552 8849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "570c763f2fd74d8da48b0bec7613fefe"
format_stamp: "Formatted at 2026-08-11 11:50:23 on dist-test-slave-bndk"
server_key: "8dec5a0ac31c6f421429bb9ba3b36bb3"
server_key_iv: "bdf5d0f9beb6739a1af9c2ae285e0341"
server_key_version: "encryptionkey@0"
I20260811 11:50:23.529551 8849 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 11:50:23.532184 8869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:23.533656 8849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:50:23.533784 8849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "570c763f2fd74d8da48b0bec7613fefe"
format_stamp: "Formatted at 2026-08-11 11:50:23 on dist-test-slave-bndk"
server_key: "8dec5a0ac31c6f421429bb9ba3b36bb3"
server_key_iv: "bdf5d0f9beb6739a1af9c2ae285e0341"
server_key_version: "encryptionkey@0"
I20260811 11:50:23.533898 8849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:23.551925 8849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:23.555371 8849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:23.555576 8849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:23.563092 8849 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:37439
I20260811 11:50:23.563391 8959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:37439 every 8 connection(s)
I20260811 11:50:23.564177 8849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:50:23.567273 8960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:23.573168 8960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe: Bootstrap starting.
I20260811 11:50:23.574340 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 8849
I20260811 11:50:23.574458 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:50:23.574725 3957 external_mini_cluster.cc:1428] Setting key a7c67020e93645683e0391b189994199
I20260811 11:50:23.576100 8960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:23.576881 8960 log.cc:824] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:23.579164 8960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe: No bootstrap required, opened a new log
I20260811 11:50:23.583683 8960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570c763f2fd74d8da48b0bec7613fefe" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 37439 } }
I20260811 11:50:23.583950 8960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:23.584000 8960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 570c763f2fd74d8da48b0bec7613fefe, State: Initialized, Role: FOLLOWER
Aug 11 11:50:23 dist-test-slave-bndk krb5kdc[8834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:23.584489 8960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570c763f2fd74d8da48b0bec7613fefe" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 37439 } }
I20260811 11:50:23.584635 8960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:50:23.584707 8960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:50:23.584820 8960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:23.586123 8960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570c763f2fd74d8da48b0bec7613fefe" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 37439 } }
I20260811 11:50:23.586546 8960 leader_election.cc:302] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 570c763f2fd74d8da48b0bec7613fefe; no voters:
I20260811 11:50:23.586925 8960 leader_election.cc:288] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:50:23.587074 8970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:23.587373 8970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe [term 1 LEADER]: Becoming Leader. State: Replica: 570c763f2fd74d8da48b0bec7613fefe, State: Running, Role: LEADER
I20260811 11:50:23.587795 8970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570c763f2fd74d8da48b0bec7613fefe" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 37439 } }
I20260811 11:50:23.588270 8960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:50:23.589175 8970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 570c763f2fd74d8da48b0bec7613fefe. Latest consensus state: current_term: 1 leader_uuid: "570c763f2fd74d8da48b0bec7613fefe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570c763f2fd74d8da48b0bec7613fefe" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 37439 } } }
I20260811 11:50:23.589422 8970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:23.589612 8972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "570c763f2fd74d8da48b0bec7613fefe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570c763f2fd74d8da48b0bec7613fefe" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 37439 } } }
I20260811 11:50:23.589713 8972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:23.589879 8985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:50:23.590699 8963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:23.575917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35692 (local address 127.3.221.126:37439)
0811 11:50:23.576427 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:23.576444 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:23.576483 (+ 39us) server_negotiation.cc:407] Connection header received
0811 11:50:23.577183 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:23.577203 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:23.577602 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:23.577901 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:23.578811 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:23.579896 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:23.581129 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:23.581747 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:23.581888 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:23.584833 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:23.584841 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:23.584861 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:23.584903 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:23.586858 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:23.587476 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:23.587483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:23.587490 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:23.587644 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:23.587988 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:23.587991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:23.587993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:23.588544 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:23.588724 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:23.589263 (+ 539us) server_negotiation.cc:299] Negotiation successful
0811 11:50:23.589619 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":145,"threads_started":1}
I20260811 11:50:23.594458 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:50:23.595199 8985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:50:23.601596 8985 catalog_manager.cc:1347] Generated new cluster ID: 0fa3a66346c54830b5792b9848400bf5
I20260811 11:50:23.601673 8985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:50:23.617712 8963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:23.611695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35706 (local address 127.3.221.126:37439)
0811 11:50:23.611820 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:23.611824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:23.612136 (+ 312us) server_negotiation.cc:407] Connection header received
0811 11:50:23.612249 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:23.612253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:23.612315 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:23.612405 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:23.613037 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:23.613883 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:23.615033 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:23.615396 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:23.615504 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:23.616482 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:23.616487 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:23.616490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:50:23.617144 (+ 654us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:23.617185 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:50:23.617207 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:23.617351 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:23.617474 (+ 123us) server_negotiation.cc:299] Negotiation successful
0811 11:50:23.617548 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 11:50:23.622743 8985 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:50:23.624123 8985 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:50:23.634379 8985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 570c763f2fd74d8da48b0bec7613fefe: Generated new TSK 0
I20260811 11:50:23.635344 8985 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:50:23.643343 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 8849
2026-08-11T11:50:23Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1585 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:50:23 dist-test-slave-bndk krb5kdc[9017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:50:23 dist-test-slave-bndk krb5kdc[9017](info): set up 2 sockets
Aug 11 11:50:23 dist-test-slave-bndk krb5kdc[9017](info): commencing operation
krb5kdc: starting...
W20260811 11:50:24.814169 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:50:24 dist-test-slave-bndk krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:50:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:50:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:50:24.945461 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:38369
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:36679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:38369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:25.061542 9046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:25.061848 9046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:25.061910 9046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:25.066134 9046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:50:25.066215 9046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:25.066243 9046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:25.066265 9046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:50:25.066284 9046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:50:25.071899 9046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:36679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:38369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:38369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:25.073063 9046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:25.074172 9046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:25.083050 9055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:25.083050 9051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:25.083053 9052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:25.083135 9046 server_base.cc:1034] running on GCE node
I20260811 11:50:25.083993 9046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:25.085248 9046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:25.086458 9046 hybrid_clock.cc:648] HybridClock initialized: now 1786449025086437 us; error 36 us; skew 500 ppm
Aug 11 11:50:25 dist-test-slave-bndk krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449025, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:25.092592 9046 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:50:25.093672 9046 webserver.cc:460] Webserver started at http://127.3.221.126:38459/ using document root <none> and password file <none>
I20260811 11:50:25.094252 9046 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:25.094334 9046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:25.094563 9046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:25.096873 9046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "7db598179cfe446885e79a00342a464f"
format_stamp: "Formatted at 2026-08-11 11:50:25 on dist-test-slave-bndk"
server_key: "dd017469986698664defd9b42f6b2214"
server_key_iv: "a8879bf1bcfb21e4bc5354ac943eb71e"
server_key_version: "encryptionkey@0"
I20260811 11:50:25.097581 9046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "7db598179cfe446885e79a00342a464f"
format_stamp: "Formatted at 2026-08-11 11:50:25 on dist-test-slave-bndk"
server_key: "dd017469986698664defd9b42f6b2214"
server_key_iv: "a8879bf1bcfb21e4bc5354ac943eb71e"
server_key_version: "encryptionkey@0"
I20260811 11:50:25.101573 9046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:50:25.104211 9066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:25.105556 9046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:50:25.105751 9046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "7db598179cfe446885e79a00342a464f"
format_stamp: "Formatted at 2026-08-11 11:50:25 on dist-test-slave-bndk"
server_key: "dd017469986698664defd9b42f6b2214"
server_key_iv: "a8879bf1bcfb21e4bc5354ac943eb71e"
server_key_version: "encryptionkey@0"
I20260811 11:50:25.105880 9046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:25.123790 9046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:25.127346 9046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:25.127642 9046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:25.135133 9046 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:38369
I20260811 11:50:25.135403 9149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:38369 every 8 connection(s)
I20260811 11:50:25.136212 9046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:50:25.139382 9151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:25.143167 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 9046
I20260811 11:50:25.143290 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:50:25.143551 3957 external_mini_cluster.cc:1428] Setting key f72b5e43b24cb24c67c5f39e0541083e
I20260811 11:50:25.146291 9151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f: Bootstrap starting.
I20260811 11:50:25.149267 9151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:25.150178 9151 log.cc:824] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:25.152565 9151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f: No bootstrap required, opened a new log
Aug 11 11:50:25 dist-test-slave-bndk krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:25.156667 9151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db598179cfe446885e79a00342a464f" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38369 } }
I20260811 11:50:25.156986 9151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:25.157055 9151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db598179cfe446885e79a00342a464f, State: Initialized, Role: FOLLOWER
I20260811 11:50:25.157620 9151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db598179cfe446885e79a00342a464f" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38369 } }
I20260811 11:50:25.157788 9151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:50:25.157859 9151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:50:25.157960 9151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:25.159184 9151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db598179cfe446885e79a00342a464f" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38369 } }
I20260811 11:50:25.159551 9151 leader_election.cc:302] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7db598179cfe446885e79a00342a464f; no voters:
I20260811 11:50:25.159881 9151 leader_election.cc:288] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:50:25.159981 9162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:25.160163 9162 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f [term 1 LEADER]: Becoming Leader. State: Replica: 7db598179cfe446885e79a00342a464f, State: Running, Role: LEADER
I20260811 11:50:25.160355 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:25.145095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51336 (local address 127.3.221.126:38369)
0811 11:50:25.145593 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:25.145613 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:25.145658 (+ 45us) server_negotiation.cc:407] Connection header received
0811 11:50:25.146507 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:25.146533 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:25.146979 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:25.147355 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:25.148060 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:25.149591 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:25.150579 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:25.151128 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:25.151309 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:25.153833 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:25.153840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:25.153860 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:25.153895 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:25.156754 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:25.157365 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:25.157372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:25.157378 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:25.157531 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:25.157866 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:25.157870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:25.157872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:25.158372 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:25.158551 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:25.158889 (+ 338us) server_negotiation.cc:299] Negotiation successful
0811 11:50:25.159212 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":165,"threads_started":1}
I20260811 11:50:25.160568 9162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db598179cfe446885e79a00342a464f" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38369 } }
I20260811 11:50:25.161446 9151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:50:25.162426 9164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7db598179cfe446885e79a00342a464f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db598179cfe446885e79a00342a464f" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38369 } } }
I20260811 11:50:25.162580 9164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:25.162925 9165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7db598179cfe446885e79a00342a464f. Latest consensus state: current_term: 1 leader_uuid: "7db598179cfe446885e79a00342a464f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db598179cfe446885e79a00342a464f" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38369 } } }
I20260811 11:50:25.163021 9165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f [sys.catalog]: This master's current role is: LEADER
W20260811 11:50:25.166628 9188 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:50:25.166712 9188 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:50:25.166795 9181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:50:25.167557 9181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:50:25.173203 9181 catalog_manager.cc:1347] Generated new cluster ID: 28b8afae4db543c9b98f235dc8676193
I20260811 11:50:25.173345 9181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:50:25.181665 9181 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:50:25.182600 9181 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:50:25.195686 9181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7db598179cfe446885e79a00342a464f: Generated new TSK 0
I20260811 11:50:25.196470 9181 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
I20260811 11:50:25.232756 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:38369
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:36679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:25.349947 9197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:25.350236 9197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:25.350281 9197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:25.354391 9197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:25.354467 9197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:25.354528 9197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:50:25.359959 9197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:36679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:38369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:25.361158 9197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:25.362283 9197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:25.372213 9204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:25.372287 9209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:25.372494 9197 server_base.cc:1034] running on GCE node
W20260811 11:50:25.372597 9202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:25.373049 9197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:25.373759 9197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:25.374938 9197 hybrid_clock.cc:648] HybridClock initialized: now 1786449025374915 us; error 30 us; skew 500 ppm
Aug 11 11:50:25 dist-test-slave-bndk krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449025, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:25.377893 9197 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:50:25.378808 9197 webserver.cc:460] Webserver started at http://127.3.221.65:33345/ using document root <none> and password file <none>
I20260811 11:50:25.379528 9197 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:25.379628 9197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:25.379886 9197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:25.382057 9197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "98c9734231c346c4a2e652371896df40"
format_stamp: "Formatted at 2026-08-11 11:50:25 on dist-test-slave-bndk"
server_key: "a773e35efdeb182ef1f05b0601addb8d"
server_key_iv: "56a3915f215695495519ca19def0367d"
server_key_version: "encryptionkey@0"
I20260811 11:50:25.382599 9197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "98c9734231c346c4a2e652371896df40"
format_stamp: "Formatted at 2026-08-11 11:50:25 on dist-test-slave-bndk"
server_key: "a773e35efdeb182ef1f05b0601addb8d"
server_key_iv: "56a3915f215695495519ca19def0367d"
server_key_version: "encryptionkey@0"
I20260811 11:50:25.386431 9197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260811 11:50:25.389077 9226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:25.390439 9197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:50:25.390552 9197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "98c9734231c346c4a2e652371896df40"
format_stamp: "Formatted at 2026-08-11 11:50:25 on dist-test-slave-bndk"
server_key: "a773e35efdeb182ef1f05b0601addb8d"
server_key_iv: "56a3915f215695495519ca19def0367d"
server_key_version: "encryptionkey@0"
I20260811 11:50:25.390650 9197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:25.406759 9197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:25.409759 9197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:25.409929 9197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:25.410571 9197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:25.411468 9197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:25.411516 9197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:25.411559 9197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:25.411580 9197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:25.420145 9197 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:35383
I20260811 11:50:25.420178 9416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:35383 every 8 connection(s)
I20260811 11:50:25.421100 9197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:50:25.430724 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 9197
Aug 11 11:50:25 dist-test-slave-bndk krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449025, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:25.430869 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:50:25.431192 3957 external_mini_cluster.cc:1428] Setting key 8d59c974d7c13204dbda712c2b87f1a7
I20260811 11:50:25.435133 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:25.422945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:38319 (local address 127.3.221.126:38369)
0811 11:50:25.423067 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:25.423071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:25.423953 (+ 882us) server_negotiation.cc:407] Connection header received
0811 11:50:25.425370 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:25.425374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:25.425441 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:25.425514 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:25.426800 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:25.427347 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:25.428487 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:25.428714 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:25.428803 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:25.431579 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:25.431583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:25.431585 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:25.431613 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:25.432980 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:25.433547 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:25.433550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:25.433551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:25.433633 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:25.433969 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:25.433971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:25.433972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:25.434150 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:25.434237 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:25.434893 (+ 656us) server_negotiation.cc:299] Negotiation successful
0811 11:50:25.435009 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 11:50:25.436378 9419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:25.423329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:38369 (local address 127.3.221.65:38319)
0811 11:50:25.423795 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:25.423840 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:25.425124 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:25.425703 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:25.425711 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:25.426099 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:25.426643 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:25.426657 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:25.427475 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:25.427478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:25.428364 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:25.428374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:25.428868 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:25.428872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:25.429036 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:25.429703 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:25.429723 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:25.431387 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:25.433092 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:25.433099 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:25.433115 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:25.433451 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:25.433717 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:25.433720 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:25.433722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:25.433873 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:25.434250 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:25.434258 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:25.434602 (+ 344us) client_negotiation.cc:769] Sending connection context
0811 11:50:25.434812 (+ 210us) client_negotiation.cc:240] Negotiation successful
0811 11:50:25.435179 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":119,"threads_started":1}
I20260811 11:50:25.437726 9417 heartbeater.cc:347] Connected to a master server at 127.3.221.126:38369
I20260811 11:50:25.437973 9417 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:25.438491 9417 heartbeater.cc:511] Master 127.3.221.126:38369 requested a full tablet report, sending...
I20260811 11:50:25.440053 9096 ts_manager.cc:194] Registered new tserver with Master: 98c9734231c346c4a2e652371896df40 (127.3.221.65:35383)
I20260811 11:50:25.441177 9096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:38319
WARNING: no policy specified for kudu/127.3.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
I20260811 11:50:25.460906 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:38369
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:36679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:25.577819 9422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:25.578078 9422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:25.578121 9422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:25.582149 9422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:25.582212 9422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:25.582271 9422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:50:25.587518 9422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:36679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:38369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:25.588557 9422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:25.589617 9422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:25.599363 9430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:25.599397 9429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:25.599363 9435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:25.599589 9422 server_base.cc:1034] running on GCE node
I20260811 11:50:25.600072 9422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:25.600732 9422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:25.601887 9422 hybrid_clock.cc:648] HybridClock initialized: now 1786449025601870 us; error 31 us; skew 500 ppm
Aug 11 11:50:25 dist-test-slave-bndk krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449025, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:25.604681 9422 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:50:25.605682 9422 webserver.cc:460] Webserver started at http://127.3.221.66:45273/ using document root <none> and password file <none>
I20260811 11:50:25.606240 9422 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:25.606309 9422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:25.606518 9422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:25.608623 9422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "8286c598c63f4a6e9560d314baa89d37"
format_stamp: "Formatted at 2026-08-11 11:50:25 on dist-test-slave-bndk"
server_key: "4dfb8c6ff6f45b7b78c73e29bb4da8d6"
server_key_iv: "31d04f4df97925159af3011036ea8f71"
server_key_version: "encryptionkey@0"
I20260811 11:50:25.609172 9422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "8286c598c63f4a6e9560d314baa89d37"
format_stamp: "Formatted at 2026-08-11 11:50:25 on dist-test-slave-bndk"
server_key: "4dfb8c6ff6f45b7b78c73e29bb4da8d6"
server_key_iv: "31d04f4df97925159af3011036ea8f71"
server_key_version: "encryptionkey@0"
I20260811 11:50:25.613157 9422 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260811 11:50:25.615680 9444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:25.616941 9422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260811 11:50:25.617071 9422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "8286c598c63f4a6e9560d314baa89d37"
format_stamp: "Formatted at 2026-08-11 11:50:25 on dist-test-slave-bndk"
server_key: "4dfb8c6ff6f45b7b78c73e29bb4da8d6"
server_key_iv: "31d04f4df97925159af3011036ea8f71"
server_key_version: "encryptionkey@0"
I20260811 11:50:25.617177 9422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:25.641739 9422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:25.644616 9422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:25.644809 9422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:25.645464 9422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:25.646309 9422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:25.646356 9422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:25.646411 9422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:25.646441 9422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:25.655287 9422 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:42075
I20260811 11:50:25.655356 9629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:42075 every 8 connection(s)
I20260811 11:50:25.656234 9422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:50:25.658694 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 9422
I20260811 11:50:25.658829 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:50:25.659114 3957 external_mini_cluster.cc:1428] Setting key 67d1a645dcde715152ed1403916782fc
Aug 11 11:50:25 dist-test-slave-bndk krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449025, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.221.67@KRBTEST.COM; defaulting to no policy
I20260811 11:50:25.671273 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:25.658126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:51499 (local address 127.3.221.126:38369)
0811 11:50:25.658259 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:25.658263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:25.659227 (+ 964us) server_negotiation.cc:407] Connection header received
0811 11:50:25.660685 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:25.660688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:25.660740 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:25.660832 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:25.662068 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:25.662787 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:25.664350 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:25.664576 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:25.664665 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:25.667469 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:25.667472 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:25.667474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:25.667501 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:25.669015 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:25.669570 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:25.669574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:25.669576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:25.669669 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:25.670021 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:25.670024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:25.670026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:25.670220 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:25.670352 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:25.670939 (+ 587us) server_negotiation.cc:299] Negotiation successful
0811 11:50:25.671076 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 11:50:25.672221 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:25.658519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:38369 (local address 127.3.221.66:51499)
0811 11:50:25.659008 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:25.659060 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:25.660486 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:25.660981 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:25.660988 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:25.661396 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:25.661931 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:25.661948 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:25.662914 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:25.662917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:25.664233 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:25.664245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:25.664722 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:25.664726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:25.664906 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:25.665564 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:25.665583 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:25.667283 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:25.669139 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:25.669146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:25.669163 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:25.669473 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:25.669757 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:25.669768 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:25.669770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:25.669914 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:25.670351 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:25.670359 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:25.670653 (+ 294us) client_negotiation.cc:769] Sending connection context
0811 11:50:25.670861 (+ 208us) client_negotiation.cc:240] Negotiation successful
0811 11:50:25.671210 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.67@KRBTEST.COM" created.
I20260811 11:50:25.673557 9630 heartbeater.cc:347] Connected to a master server at 127.3.221.126:38369
I20260811 11:50:25.673840 9630 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:25.674461 9630 heartbeater.cc:511] Master 127.3.221.126:38369 requested a full tablet report, sending...
I20260811 11:50:25.675573 9096 ts_manager.cc:194] Registered new tserver with Master: 8286c598c63f4a6e9560d314baa89d37 (127.3.221.66:42075)
I20260811 11:50:25.676299 9096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:51499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
I20260811 11:50:25.689522 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:38369
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:36679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:25.808024 9637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:25.808292 9637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:25.808339 9637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:25.812420 9637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:25.812486 9637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:25.812552 9637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:50:25.818037 9637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:36679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:38369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:25.819125 9637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:25.820194 9637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:25.830109 9646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:25.830130 9653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:25.830111 9648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:25.830377 9637 server_base.cc:1034] running on GCE node
I20260811 11:50:25.830803 9637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:25.831478 9637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:25.832659 9637 hybrid_clock.cc:648] HybridClock initialized: now 1786449025832637 us; error 32 us; skew 500 ppm
Aug 11 11:50:25 dist-test-slave-bndk krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449025, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:25.835520 9637 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:50:25.836453 9637 webserver.cc:460] Webserver started at http://127.3.221.67:44447/ using document root <none> and password file <none>
I20260811 11:50:25.837034 9637 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:25.837105 9637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:25.837390 9637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:25.839890 9637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "db0b73276ad14e7c84989485c57c3c07"
format_stamp: "Formatted at 2026-08-11 11:50:25 on dist-test-slave-bndk"
server_key: "f3120d1a68dc82e7a9d0b9fd1b469478"
server_key_iv: "baf3c6978b9e418b0e0d062dd1db0337"
server_key_version: "encryptionkey@0"
I20260811 11:50:25.840610 9637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "db0b73276ad14e7c84989485c57c3c07"
format_stamp: "Formatted at 2026-08-11 11:50:25 on dist-test-slave-bndk"
server_key: "f3120d1a68dc82e7a9d0b9fd1b469478"
server_key_iv: "baf3c6978b9e418b0e0d062dd1db0337"
server_key_version: "encryptionkey@0"
I20260811 11:50:25.844661 9637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260811 11:50:25.847157 9666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:25.848416 9637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 11:50:25.848531 9637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "db0b73276ad14e7c84989485c57c3c07"
format_stamp: "Formatted at 2026-08-11 11:50:25 on dist-test-slave-bndk"
server_key: "f3120d1a68dc82e7a9d0b9fd1b469478"
server_key_iv: "baf3c6978b9e418b0e0d062dd1db0337"
server_key_version: "encryptionkey@0"
I20260811 11:50:25.848630 9637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:25.865350 9637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:25.868424 9637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:25.868606 9637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:25.869261 9637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:25.870137 9637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:25.870184 9637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:25.870236 9637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:25.870265 9637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:25.878931 9637 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:38153
I20260811 11:50:25.878947 9851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:38153 every 8 connection(s)
I20260811 11:50:25.879858 9637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:50:25.888342 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 9637
I20260811 11:50:25.888499 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:50:25.888840 3957 external_mini_cluster.cc:1428] Setting key d938273042f6a8cd83fa93d7316cbe52
Aug 11 11:50:25 dist-test-slave-bndk krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449025, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:25.893858 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:25.881844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:38155 (local address 127.3.221.126:38369)
0811 11:50:25.881974 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:25.881978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:25.882835 (+ 857us) server_negotiation.cc:407] Connection header received
0811 11:50:25.884145 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:25.884148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:25.884209 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:25.884283 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:25.885521 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:25.886070 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:25.887197 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:25.887420 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:25.887512 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:25.890399 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:25.890401 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:25.890403 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:25.890430 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:25.891779 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:25.892292 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:25.892294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:25.892296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:25.892374 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:25.892696 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:25.892699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:25.892701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:25.892897 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:25.892984 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:25.893551 (+ 567us) server_negotiation.cc:299] Negotiation successful
0811 11:50:25.893681 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 11:50:25.894747 9856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:25.882219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:38369 (local address 127.3.221.67:38155)
0811 11:50:25.882672 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:25.882716 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:25.883935 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:25.884472 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:25.884479 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:25.884844 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:25.885379 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:25.885396 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:25.886192 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:25.886195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:25.887074 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:25.887084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:25.887571 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:25.887575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:25.887745 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:25.888358 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:25.888380 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:25.890224 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:25.891895 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:25.891902 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:25.891921 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:25.892191 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:25.892458 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:25.892461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:25.892463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:25.892608 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:25.892997 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:25.893004 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:25.893316 (+ 312us) client_negotiation.cc:769] Sending connection context
0811 11:50:25.893495 (+ 179us) client_negotiation.cc:240] Negotiation successful
0811 11:50:25.893843 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":116,"threads_started":1}
I20260811 11:50:25.895936 9853 heartbeater.cc:347] Connected to a master server at 127.3.221.126:38369
I20260811 11:50:25.896190 9853 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:25.896695 9853 heartbeater.cc:511] Master 127.3.221.126:38369 requested a full tablet report, sending...
I20260811 11:50:25.897857 9096 ts_manager.cc:194] Registered new tserver with Master: db0b73276ad14e7c84989485c57c3c07 (127.3.221.67:38153)
I20260811 11:50:25.898445 9096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:38155
I20260811 11:50:25.902733 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 11:50:25 dist-test-slave-bndk krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449025, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 11 11:50:25 dist-test-slave-bndk krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449025, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:25.930336 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:25.921748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51350 (local address 127.3.221.126:38369)
0811 11:50:25.921886 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:25.921889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:25.922008 (+ 119us) server_negotiation.cc:407] Connection header received
0811 11:50:25.922183 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:25.922186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:25.922237 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:25.922305 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:25.923039 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:25.923565 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:25.924409 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:25.924622 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:25.924710 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:25.927238 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:25.927242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:25.927245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:25.927275 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:25.928590 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:25.929022 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:25.929025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:25.929026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:25.929112 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:25.929415 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:25.929419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:25.929421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:25.929662 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:25.929798 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:25.930044 (+ 246us) server_negotiation.cc:299] Negotiation successful
0811 11:50:25.930170 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260811 11:50:25.930769 9096 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51350
I20260811 11:50:25.931645 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 9197
I20260811 11:50:25.945380 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 9422
I20260811 11:50:25.959867 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 9637
I20260811 11:50:25.971546 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 9046
2026-08-11T11:50:25Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2322 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:50:26 dist-test-slave-bndk krb5kdc[9871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:50:26 dist-test-slave-bndk krb5kdc[9871](info): set up 2 sockets
Aug 11 11:50:26 dist-test-slave-bndk krb5kdc[9871](info): commencing operation
krb5kdc: starting...
W20260811 11:50:27.148947 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:50:27 dist-test-slave-bndk krb5kdc[9871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:50:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:50:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:50:27.278322 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:39115
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:46257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:39115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:27.394965 9894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:27.395283 9894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:27.395356 9894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:27.399585 9894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:50:27.399670 9894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:27.399698 9894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:27.399720 9894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:50:27.399750 9894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:50:27.405398 9894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:46257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:39115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:39115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:27.406558 9894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:27.407625 9894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:27.416358 9905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:27.416344 9908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:27.416414 9906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:27.416517 9894 server_base.cc:1034] running on GCE node
I20260811 11:50:27.417312 9894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:27.418533 9894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:27.419703 9894 hybrid_clock.cc:648] HybridClock initialized: now 1786449027419691 us; error 34 us; skew 500 ppm
Aug 11 11:50:27 dist-test-slave-bndk krb5kdc[9871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449027, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:27.422580 9894 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:50:27.423548 9894 webserver.cc:460] Webserver started at http://127.3.221.126:46291/ using document root <none> and password file <none>
I20260811 11:50:27.424114 9894 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:27.424180 9894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:27.424386 9894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:27.426575 9894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "258cb62d42bf43608f903478ef6ecae0"
format_stamp: "Formatted at 2026-08-11 11:50:27 on dist-test-slave-bndk"
server_key: "22b198f66794da180dd4685fdaa8ff61"
server_key_iv: "9e6c207ef3e20e8e5c412e05b29c94fb"
server_key_version: "encryptionkey@0"
I20260811 11:50:27.427163 9894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "258cb62d42bf43608f903478ef6ecae0"
format_stamp: "Formatted at 2026-08-11 11:50:27 on dist-test-slave-bndk"
server_key: "22b198f66794da180dd4685fdaa8ff61"
server_key_iv: "9e6c207ef3e20e8e5c412e05b29c94fb"
server_key_version: "encryptionkey@0"
I20260811 11:50:27.431202 9894 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 11:50:27.433815 9920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:27.435122 9894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 11:50:27.435281 9894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "258cb62d42bf43608f903478ef6ecae0"
format_stamp: "Formatted at 2026-08-11 11:50:27 on dist-test-slave-bndk"
server_key: "22b198f66794da180dd4685fdaa8ff61"
server_key_iv: "9e6c207ef3e20e8e5c412e05b29c94fb"
server_key_version: "encryptionkey@0"
I20260811 11:50:27.435401 9894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:27.449281 9894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:27.452301 9894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:27.452484 9894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:27.459811 9894 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:39115
I20260811 11:50:27.459825 10006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:39115 every 8 connection(s)
I20260811 11:50:27.460819 9894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:50:27.463656 10007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:27.465864 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 9894
I20260811 11:50:27.466008 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:50:27.466308 3957 external_mini_cluster.cc:1428] Setting key 089bb2dc4dbef03227fe4275f082d54b
I20260811 11:50:27.470201 10007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0: Bootstrap starting.
I20260811 11:50:27.472536 10007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:27.473469 10007 log.cc:824] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0: Log is configured to *not* fsync() on all Append() calls
Aug 11 11:50:27 dist-test-slave-bndk krb5kdc[9871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:27.475703 10007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0: No bootstrap required, opened a new log
I20260811 11:50:27.480424 10007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258cb62d42bf43608f903478ef6ecae0" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 39115 } }
I20260811 11:50:27.480718 10007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:27.480789 10007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 258cb62d42bf43608f903478ef6ecae0, State: Initialized, Role: FOLLOWER
I20260811 11:50:27.481307 10007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258cb62d42bf43608f903478ef6ecae0" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 39115 } }
I20260811 11:50:27.481436 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:27.467888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51640 (local address 127.3.221.126:39115)
0811 11:50:27.468410 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:27.468426 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:27.468466 (+ 40us) server_negotiation.cc:407] Connection header received
0811 11:50:27.469291 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:27.469333 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:27.469685 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:27.470012 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:27.470843 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:27.471873 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:27.472713 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:27.473458 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:27.473635 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:27.476145 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:27.476153 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:27.476171 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:27.476202 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:27.478149 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:27.478564 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:27.478570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:27.478576 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:27.478722 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:27.478955 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:27.478958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:27.478959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:27.479505 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:27.479711 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:27.480028 (+ 317us) server_negotiation.cc:299] Negotiation successful
0811 11:50:27.480354 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":133,"threads_started":1}
I20260811 11:50:27.481487 10007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:50:27.481544 10007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:50:27.481643 10007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:27.482632 10007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258cb62d42bf43608f903478ef6ecae0" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 39115 } }
I20260811 11:50:27.483008 10007 leader_election.cc:302] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 258cb62d42bf43608f903478ef6ecae0; no voters:
I20260811 11:50:27.483352 10007 leader_election.cc:288] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:50:27.483466 10018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:27.483693 10018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0 [term 1 LEADER]: Becoming Leader. State: Replica: 258cb62d42bf43608f903478ef6ecae0, State: Running, Role: LEADER
I20260811 11:50:27.484037 10018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258cb62d42bf43608f903478ef6ecae0" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 39115 } }
I20260811 11:50:27.484622 10007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:50:27.485270 10018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 258cb62d42bf43608f903478ef6ecae0. Latest consensus state: current_term: 1 leader_uuid: "258cb62d42bf43608f903478ef6ecae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258cb62d42bf43608f903478ef6ecae0" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 39115 } } }
I20260811 11:50:27.485389 10018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:27.485689 10019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "258cb62d42bf43608f903478ef6ecae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258cb62d42bf43608f903478ef6ecae0" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 39115 } } }
I20260811 11:50:27.485834 10019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:27.486702 10028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:50:27.490072 10028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:50:27.496017 10028 catalog_manager.cc:1347] Generated new cluster ID: 1b33f885b9064ad4b9ca8fb317fe608b
I20260811 11:50:27.496100 10028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:50:27.504947 10028 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:50:27.506292 10028 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:50:27.517248 10028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 258cb62d42bf43608f903478ef6ecae0: Generated new TSK 0
I20260811 11:50:27.518169 10028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
I20260811 11:50:27.552381 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:39115
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:46257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:27.669106 10055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:27.669453 10055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:27.669517 10055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:27.673669 10055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:27.673741 10055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:27.673837 10055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:50:27.679329 10055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:46257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:39115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:27.680481 10055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:27.681602 10055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:27.691576 10067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:27.691701 10068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:27.691833 10055 server_base.cc:1034] running on GCE node
W20260811 11:50:27.691560 10072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:27.692457 10055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:27.693311 10055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:27.694504 10055 hybrid_clock.cc:648] HybridClock initialized: now 1786449027694474 us; error 41 us; skew 500 ppm
Aug 11 11:50:27 dist-test-slave-bndk krb5kdc[9871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449027, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:27.698228 10055 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:50:27.699432 10055 webserver.cc:460] Webserver started at http://127.3.221.65:40459/ using document root <none> and password file <none>
I20260811 11:50:27.700033 10055 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:27.700104 10055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:27.700314 10055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:27.702512 10055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "f3f82d133a2a48dda5fd87088881f754"
format_stamp: "Formatted at 2026-08-11 11:50:27 on dist-test-slave-bndk"
server_key: "d8c5b29934b5ceaf08dd71c65408ced3"
server_key_iv: "bdef67a68aeb82cdc728ba8740ea9830"
server_key_version: "encryptionkey@0"
I20260811 11:50:27.703091 10055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "f3f82d133a2a48dda5fd87088881f754"
format_stamp: "Formatted at 2026-08-11 11:50:27 on dist-test-slave-bndk"
server_key: "d8c5b29934b5ceaf08dd71c65408ced3"
server_key_iv: "bdef67a68aeb82cdc728ba8740ea9830"
server_key_version: "encryptionkey@0"
I20260811 11:50:27.707198 10055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 11:50:27.709867 10084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:27.711139 10055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:50:27.711272 10055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "f3f82d133a2a48dda5fd87088881f754"
format_stamp: "Formatted at 2026-08-11 11:50:27 on dist-test-slave-bndk"
server_key: "d8c5b29934b5ceaf08dd71c65408ced3"
server_key_iv: "bdef67a68aeb82cdc728ba8740ea9830"
server_key_version: "encryptionkey@0"
I20260811 11:50:27.711380 10055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:27.721772 10055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:27.724959 10055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:27.725147 10055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:27.725788 10055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:27.726639 10055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:27.726686 10055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:27.726747 10055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:27.726774 10055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:27.735186 10055 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:33503
I20260811 11:50:27.735213 10246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:33503 every 8 connection(s)
I20260811 11:50:27.736114 10055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:50:27.740124 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 10055
I20260811 11:50:27.740248 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:50:27.740532 3957 external_mini_cluster.cc:1428] Setting key f2ef98b31e9fe48522f75bec7e22e4f9
Aug 11 11:50:27 dist-test-slave-bndk krb5kdc[9871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449027, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:27.750653 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:27.738028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:43501 (local address 127.3.221.126:39115)
0811 11:50:27.738177 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:27.738182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:27.739111 (+ 929us) server_negotiation.cc:407] Connection header received
0811 11:50:27.740614 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:27.740617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:27.740679 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:27.740758 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:27.742146 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:27.742678 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:27.743822 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:27.744069 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:27.744159 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:27.747011 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:27.747015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:27.747018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:27.747051 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:27.748406 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:27.749006 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:27.749009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:27.749010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:27.749080 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:27.749469 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:27.749472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:27.749473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:27.749654 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:27.749725 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:27.750408 (+ 683us) server_negotiation.cc:299] Negotiation successful
0811 11:50:27.750523 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.3.221.66@KRBTEST.COM; defaulting to no policy
I20260811 11:50:27.751756 10253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:27.738440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:39115 (local address 127.3.221.65:43501)
0811 11:50:27.738950 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:27.738997 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:27.740391 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:27.740960 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:27.740967 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:27.741438 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:27.742010 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:27.742026 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:27.742800 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:27.742802 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:27.743683 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:27.743695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:27.744224 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:27.744228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:27.744423 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:27.745064 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:27.745085 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:27.746818 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:27.748527 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:27.748534 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:27.748552 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:27.748906 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:27.749157 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:27.749160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:27.749161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:27.749375 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:27.749750 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:27.749757 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:27.750107 (+ 350us) client_negotiation.cc:769] Sending connection context
0811 11:50:27.750336 (+ 229us) client_negotiation.cc:240] Negotiation successful
0811 11:50:27.750709 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":125,"threads_started":1}
I20260811 11:50:27.752942 10247 heartbeater.cc:347] Connected to a master server at 127.3.221.126:39115
I20260811 11:50:27.753190 10247 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:27.753789 10247 heartbeater.cc:511] Master 127.3.221.126:39115 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.66@KRBTEST.COM" created.
I20260811 11:50:27.755579 9948 ts_manager.cc:194] Registered new tserver with Master: f3f82d133a2a48dda5fd87088881f754 (127.3.221.65:33503)
I20260811 11:50:27.756824 9948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:43501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
I20260811 11:50:27.772409 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:39115
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:46257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:27.889406 10256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:27.889680 10256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:27.889743 10256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:27.893914 10256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:27.893991 10256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:27.894078 10256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:50:27.899750 10256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:46257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:39115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:27.900872 10256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:27.901969 10256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:27.911679 10273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:27.911688 10268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:27.911732 10256 server_base.cc:1034] running on GCE node
W20260811 11:50:27.911688 10269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:27.912261 10256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:27.912928 10256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:27.914094 10256 hybrid_clock.cc:648] HybridClock initialized: now 1786449027914058 us; error 50 us; skew 500 ppm
Aug 11 11:50:27 dist-test-slave-bndk krb5kdc[9871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449027, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:27.916863 10256 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:50:27.917821 10256 webserver.cc:460] Webserver started at http://127.3.221.66:36953/ using document root <none> and password file <none>
I20260811 11:50:27.918368 10256 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:27.918440 10256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:27.918658 10256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:27.920753 10256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "f8b4403ee97843518d0e7b2d803499fe"
format_stamp: "Formatted at 2026-08-11 11:50:27 on dist-test-slave-bndk"
server_key: "8f5cb060769e7e16929ad95d48ec4634"
server_key_iv: "4e3173adbdef6ace30140847b872ea84"
server_key_version: "encryptionkey@0"
I20260811 11:50:27.921342 10256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "f8b4403ee97843518d0e7b2d803499fe"
format_stamp: "Formatted at 2026-08-11 11:50:27 on dist-test-slave-bndk"
server_key: "8f5cb060769e7e16929ad95d48ec4634"
server_key_iv: "4e3173adbdef6ace30140847b872ea84"
server_key_version: "encryptionkey@0"
I20260811 11:50:27.925017 10256 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:50:27.927551 10290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:27.928781 10256 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 11:50:27.928895 10256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "f8b4403ee97843518d0e7b2d803499fe"
format_stamp: "Formatted at 2026-08-11 11:50:27 on dist-test-slave-bndk"
server_key: "8f5cb060769e7e16929ad95d48ec4634"
server_key_iv: "4e3173adbdef6ace30140847b872ea84"
server_key_version: "encryptionkey@0"
I20260811 11:50:27.928999 10256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:27.952076 10256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:27.955264 10256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:27.955456 10256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:27.956063 10256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:27.956928 10256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:27.956975 10256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:27.957031 10256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:27.957057 10256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:27.965973 10256 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:46175
I20260811 11:50:27.965998 10476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:46175 every 8 connection(s)
I20260811 11:50:27.966866 10256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:50:27.970402 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 10256
I20260811 11:50:27.970512 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:50:27.970772 3957 external_mini_cluster.cc:1428] Setting key a5769a4a5cb4543cb8b0f37762c66c1e
Aug 11 11:50:27 dist-test-slave-bndk krb5kdc[9871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449027, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.221.67@KRBTEST.COM; defaulting to no policy
I20260811 11:50:27.981647 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:27.968705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:39859 (local address 127.3.221.126:39115)
0811 11:50:27.968855 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:27.968860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:27.969798 (+ 938us) server_negotiation.cc:407] Connection header received
0811 11:50:27.971253 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:27.971257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:27.971309 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:27.971411 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:27.972834 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:27.973564 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:27.974910 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:27.975130 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:27.975211 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:27.978119 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:27.978123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:27.978125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:27.978153 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:27.979535 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:27.980074 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:27.980077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:27.980078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:27.980157 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:27.980504 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:27.980507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:27.980509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:27.980700 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:27.980789 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:27.981339 (+ 550us) server_negotiation.cc:299] Negotiation successful
0811 11:50:27.981474 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 11:50:27.982568 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:27.969126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:39115 (local address 127.3.221.66:39859)
0811 11:50:27.969632 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:27.969679 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:27.971008 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:27.971589 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:27.971598 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:27.972087 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:27.972682 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:27.972697 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:27.973677 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:27.973679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:27.974788 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:27.974800 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:27.975285 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:27.975289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:27.975505 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:27.976138 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:27.976159 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:27.977928 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:27.979661 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:27.979668 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:27.979687 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:27.979971 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:27.980258 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:27.980261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:27.980263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:27.980410 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:27.980806 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:27.980813 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:27.981071 (+ 258us) client_negotiation.cc:769] Sending connection context
0811 11:50:27.981271 (+ 200us) client_negotiation.cc:240] Negotiation successful
0811 11:50:27.981640 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":125,"threads_started":1}
I20260811 11:50:27.983768 10477 heartbeater.cc:347] Connected to a master server at 127.3.221.126:39115
I20260811 11:50:27.984025 10477 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.67@KRBTEST.COM" created.
I20260811 11:50:27.984607 10477 heartbeater.cc:511] Master 127.3.221.126:39115 requested a full tablet report, sending...
I20260811 11:50:27.985733 9948 ts_manager.cc:194] Registered new tserver with Master: f8b4403ee97843518d0e7b2d803499fe (127.3.221.66:46175)
I20260811 11:50:27.986531 9948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:39859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
I20260811 11:50:28.001355 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:39115
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:46257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:28.116909 10489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:28.117200 10489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:28.117292 10489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:28.121532 10489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:28.121604 10489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:28.121665 10489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:50:28.127206 10489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:46257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:39115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:28.128401 10489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:28.129529 10489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:28.139753 10499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:28.139753 10498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:28.139824 10489 server_base.cc:1034] running on GCE node
W20260811 11:50:28.139781 10501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:28.140436 10489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:28.141108 10489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:28.142297 10489 hybrid_clock.cc:648] HybridClock initialized: now 1786449028142240 us; error 69 us; skew 500 ppm
Aug 11 11:50:28 dist-test-slave-bndk krb5kdc[9871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449028, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:28.145306 10489 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:50:28.146387 10489 webserver.cc:460] Webserver started at http://127.3.221.67:36327/ using document root <none> and password file <none>
I20260811 11:50:28.146997 10489 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:28.147073 10489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:28.147292 10489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:28.149741 10489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "3540b2b4155344dab9480f77394807d1"
format_stamp: "Formatted at 2026-08-11 11:50:28 on dist-test-slave-bndk"
server_key: "063efd12777c386d46cb09b543c9c797"
server_key_iv: "ecbe49096c623022a16d8f3bbd86a969"
server_key_version: "encryptionkey@0"
I20260811 11:50:28.150379 10489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "3540b2b4155344dab9480f77394807d1"
format_stamp: "Formatted at 2026-08-11 11:50:28 on dist-test-slave-bndk"
server_key: "063efd12777c386d46cb09b543c9c797"
server_key_iv: "ecbe49096c623022a16d8f3bbd86a969"
server_key_version: "encryptionkey@0"
I20260811 11:50:28.154498 10489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260811 11:50:28.157162 10512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:28.158480 10489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 11:50:28.158605 10489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "3540b2b4155344dab9480f77394807d1"
format_stamp: "Formatted at 2026-08-11 11:50:28 on dist-test-slave-bndk"
server_key: "063efd12777c386d46cb09b543c9c797"
server_key_iv: "ecbe49096c623022a16d8f3bbd86a969"
server_key_version: "encryptionkey@0"
I20260811 11:50:28.158713 10489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:28.175006 10489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:28.178242 10489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:28.178455 10489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:28.179109 10489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:28.179994 10489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:28.180045 10489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:28.180100 10489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:28.180130 10489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:28.189414 10489 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:38969
I20260811 11:50:28.189448 10697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:38969 every 8 connection(s)
I20260811 11:50:28.190371 10489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:50:28.199016 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 10489
I20260811 11:50:28.199165 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:50:28.199466 3957 external_mini_cluster.cc:1428] Setting key 2c14d7385d5612476ce1239f69e3edbd
Aug 11 11:50:28 dist-test-slave-bndk krb5kdc[9871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449028, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:28.204457 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:28.192260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:36879 (local address 127.3.221.126:39115)
0811 11:50:28.192406 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:28.192411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:28.193337 (+ 926us) server_negotiation.cc:407] Connection header received
0811 11:50:28.194749 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:28.194753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:28.194817 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:28.194893 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:28.196157 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:28.196695 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:28.197873 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:28.198096 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:28.198189 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:28.200886 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:28.200890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:28.200893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:28.200927 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:28.202470 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:28.202984 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:28.202987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:28.202988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:28.203080 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:28.203353 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:28.203355 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:28.203356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:28.203544 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:28.203623 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:28.204177 (+ 554us) server_negotiation.cc:299] Negotiation successful
0811 11:50:28.204331 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 11:50:28.205345 10703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:28.192605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:39115 (local address 127.3.221.67:36879)
0811 11:50:28.193145 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:28.193193 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:28.194518 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:28.195091 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:28.195099 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:28.195488 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:28.196001 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:28.196016 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:28.196823 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:28.196826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:28.197749 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:28.197759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:28.198247 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:28.198251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:28.198416 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:28.199015 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:28.199033 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:28.200678 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:28.202584 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:28.202590 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:28.202605 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:28.202890 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:28.203148 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:28.203150 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:28.203152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:28.203286 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:28.203620 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:28.203626 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:28.203901 (+ 275us) client_negotiation.cc:769] Sending connection context
0811 11:50:28.204073 (+ 172us) client_negotiation.cc:240] Negotiation successful
0811 11:50:28.204356 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":135,"threads_started":1}
I20260811 11:50:28.206590 10698 heartbeater.cc:347] Connected to a master server at 127.3.221.126:39115
I20260811 11:50:28.206877 10698 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:28.207407 10698 heartbeater.cc:511] Master 127.3.221.126:39115 requested a full tablet report, sending...
I20260811 11:50:28.208530 9948 ts_manager.cc:194] Registered new tserver with Master: 3540b2b4155344dab9480f77394807d1 (127.3.221.67:38969)
I20260811 11:50:28.209139 9948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:36879
I20260811 11:50:28.213805 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 11:50:28 dist-test-slave-bndk krb5kdc[9871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 11:50:28 dist-test-slave-bndk krb5kdc[9871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.65@KRBTEST.COM
I20260811 11:50:28.240886 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:28.231502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54276 (local address 127.3.221.65:33503)
0811 11:50:28.231790 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:28.231807 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:28.231834 (+ 27us) server_negotiation.cc:407] Connection header received
0811 11:50:28.231918 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:28.231923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:28.232157 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:28.232281 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:28.232973 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:28.233930 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:28.234780 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:28.235177 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:28.235262 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:28.237616 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:28.237627 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:28.237642 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:28.237669 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:28.239176 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:28.239618 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:28.239623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:28.239624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:28.239708 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:28.239999 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:28.240001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:28.240003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:28.240255 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:28.240439 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:28.240588 (+ 149us) server_negotiation.cc:299] Negotiation successful
0811 11:50:28.240721 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":89,"threads_started":1}
Aug 11 11:50:28 dist-test-slave-bndk krb5kdc[9871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:28.251191 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:28.243778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51642 (local address 127.3.221.126:39115)
0811 11:50:28.243916 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:28.243920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:28.243961 (+ 41us) server_negotiation.cc:407] Connection header received
0811 11:50:28.244072 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:28.244074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:28.244116 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:28.244179 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:28.244735 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:28.245299 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:28.246041 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:28.246241 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:28.246347 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:28.248543 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:28.248546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:28.248548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:28.248574 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:28.249839 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:28.250189 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:28.250191 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:28.250193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:28.250259 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:28.250448 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:28.250450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:28.250451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:28.250654 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:28.250754 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:28.250930 (+ 176us) server_negotiation.cc:299] Negotiation successful
0811 11:50:28.251030 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260811 11:50:28.251639 9948 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51642
I20260811 11:50:28.252246 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 10055
I20260811 11:50:28.266234 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 10256
I20260811 11:50:28.275754 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 10489
I20260811 11:50:28.287038 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 9894
2026-08-11T11:50:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2315 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:50:28 dist-test-slave-bndk krb5kdc[10719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:50:28 dist-test-slave-bndk krb5kdc[10719](info): set up 2 sockets
Aug 11 11:50:28 dist-test-slave-bndk krb5kdc[10719](info): commencing operation
krb5kdc: starting...
W20260811 11:50:29.540973 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.224s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260811 11:50:29.699425 3957 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.158s user 0.002s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260811 11:50:29.888005 3957 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.188s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/test-user.keytab.
W20260811 11:50:30.502157 3957 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.536s user 0.000s sys 0.002s
Aug 11 11:50:30 dist-test-slave-bndk krb5kdc[10719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:50:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:50:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
W20260811 11:50:31.003605 3957 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.221.126: real 0.475s user 0.000s sys 0.004s
I20260811 11:50:31.004179 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:43013
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:43013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:31.124248 10739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:31.124547 10739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:31.124590 10739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:31.128633 10739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:50:31.128702 10739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:31.128721 10739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:31.128736 10739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:50:31.128750 10739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:50:31.134373 10739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:43013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:43013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:31.135969 10739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:31.137130 10739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:31.145864 10745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:31.145906 10739 server_base.cc:1034] running on GCE node
W20260811 11:50:31.145864 10747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:31.145864 10749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:31.146728 10739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:31.147886 10739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:31.149060 10739 hybrid_clock.cc:648] HybridClock initialized: now 1786449031149043 us; error 28 us; skew 500 ppm
Aug 11 11:50:31 dist-test-slave-bndk krb5kdc[10719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449031, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:31.157171 10739 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:50:31.158128 10739 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:31.158210 10739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:31.158447 10739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:31.160759 10739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "4ced5041a0e74ccda312f6fee6249ee5"
format_stamp: "Formatted at 2026-08-11 11:50:31 on dist-test-slave-bndk"
server_key: "8a7b960064d29a80930baef377b5a600"
server_key_iv: "87aad4ce2af6ee436e4c0bf1d0e1c6c6"
server_key_version: "encryptionkey@0"
I20260811 11:50:31.161424 10739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "4ced5041a0e74ccda312f6fee6249ee5"
format_stamp: "Formatted at 2026-08-11 11:50:31 on dist-test-slave-bndk"
server_key: "8a7b960064d29a80930baef377b5a600"
server_key_iv: "87aad4ce2af6ee436e4c0bf1d0e1c6c6"
server_key_version: "encryptionkey@0"
I20260811 11:50:31.165568 10739 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:50:31.168365 10757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:31.169872 10739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:50:31.170017 10739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "4ced5041a0e74ccda312f6fee6249ee5"
format_stamp: "Formatted at 2026-08-11 11:50:31 on dist-test-slave-bndk"
server_key: "8a7b960064d29a80930baef377b5a600"
server_key_iv: "87aad4ce2af6ee436e4c0bf1d0e1c6c6"
server_key_version: "encryptionkey@0"
I20260811 11:50:31.170135 10739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:31.191007 10739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:31.194765 10739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:31.195196 10739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:31.203127 10739 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:43013
I20260811 11:50:31.203440 10832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:43013 every 8 connection(s)
I20260811 11:50:31.203924 10739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:50:31.206904 10837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:31.212560 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 10739
I20260811 11:50:31.212702 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:50:31.212932 10837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5: Bootstrap starting.
I20260811 11:50:31.212970 3957 external_mini_cluster.cc:1428] Setting key a051bc2a4ef8b0aab92184d95d9f8c2a
I20260811 11:50:31.215600 10837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:31.216461 10837 log.cc:824] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:31.218930 10837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5: No bootstrap required, opened a new log
Aug 11 11:50:31 dist-test-slave-bndk krb5kdc[10719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:31.223467 10837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ced5041a0e74ccda312f6fee6249ee5" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 43013 } }
I20260811 11:50:31.223807 10837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:31.223881 10837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ced5041a0e74ccda312f6fee6249ee5, State: Initialized, Role: FOLLOWER
I20260811 11:50:31.224406 10837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ced5041a0e74ccda312f6fee6249ee5" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 43013 } }
I20260811 11:50:31.224547 10837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:50:31.224617 10837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:50:31.224722 10837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:31.225872 10837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ced5041a0e74ccda312f6fee6249ee5" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 43013 } }
I20260811 11:50:31.226238 10837 leader_election.cc:302] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ced5041a0e74ccda312f6fee6249ee5; no voters:
I20260811 11:50:31.226590 10837 leader_election.cc:288] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:50:31.226689 10847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:31.226861 10847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5 [term 1 LEADER]: Becoming Leader. State: Replica: 4ced5041a0e74ccda312f6fee6249ee5, State: Running, Role: LEADER
I20260811 11:50:31.227249 10847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ced5041a0e74ccda312f6fee6249ee5" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 43013 } }
I20260811 11:50:31.228029 10837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:50:31.228474 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:31.214470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44404 (local address 127.3.221.126:43013)
0811 11:50:31.214938 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:31.214956 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:31.214994 (+ 38us) server_negotiation.cc:407] Connection header received
0811 11:50:31.215702 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:31.215729 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:31.216088 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:31.216398 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:31.217329 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:31.218359 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:31.219569 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:31.220150 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:31.220323 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:31.223368 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:31.223375 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:31.223394 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:31.223431 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:31.225332 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:31.225820 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:31.225824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:31.225829 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:31.225960 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:31.226287 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:31.226291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:31.226292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:31.226793 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:31.227032 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:31.227311 (+ 279us) server_negotiation.cc:299] Negotiation successful
0811 11:50:31.227617 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":151,"threads_started":1}
I20260811 11:50:31.229156 10848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ced5041a0e74ccda312f6fee6249ee5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ced5041a0e74ccda312f6fee6249ee5" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 43013 } } }
I20260811 11:50:31.229195 10851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ced5041a0e74ccda312f6fee6249ee5. Latest consensus state: current_term: 1 leader_uuid: "4ced5041a0e74ccda312f6fee6249ee5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ced5041a0e74ccda312f6fee6249ee5" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 43013 } } }
I20260811 11:50:31.230201 10851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:31.230201 10848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5 [sys.catalog]: This master's current role is: LEADER
W20260811 11:50:31.233741 10872 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:50:31.233836 10872 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:50:31.233940 10869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:50:31.234917 10869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:50:31.240558 10869 catalog_manager.cc:1347] Generated new cluster ID: 3cf43569c50c455eaa3a0bac6cc00ea0
I20260811 11:50:31.240641 10869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:50:31.253046 10869 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:50:31.254150 10869 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:50:31.261600 10869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5: Generated new TSK 0
I20260811 11:50:31.262322 10869 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
I20260811 11:50:31.300658 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:43013
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:31.431085 10878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:31.431398 10878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:31.431447 10878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:31.435642 10878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:31.435721 10878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:31.435788 10878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:50:31.441497 10878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:43013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:31.442628 10878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:31.443692 10878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:31.453509 10890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:31.453482 10888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:31.453464 10893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:31.453552 10878 server_base.cc:1034] running on GCE node
I20260811 11:50:31.454221 10878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:31.454864 10878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:31.456035 10878 hybrid_clock.cc:648] HybridClock initialized: now 1786449031456017 us; error 32 us; skew 500 ppm
Aug 11 11:50:31 dist-test-slave-bndk krb5kdc[10719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449031, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:31.459020 10878 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:50:31.459789 10878 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:31.459874 10878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:31.460103 10878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:31.462303 10878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "5c238abf7e7b4e408709b3513fa869e1"
format_stamp: "Formatted at 2026-08-11 11:50:31 on dist-test-slave-bndk"
server_key: "1aad9dba5cc09ea4adc77ceb23a7dc0c"
server_key_iv: "25810179bc856820dce0d40e2102dc06"
server_key_version: "encryptionkey@0"
I20260811 11:50:31.462867 10878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "5c238abf7e7b4e408709b3513fa869e1"
format_stamp: "Formatted at 2026-08-11 11:50:31 on dist-test-slave-bndk"
server_key: "1aad9dba5cc09ea4adc77ceb23a7dc0c"
server_key_iv: "25810179bc856820dce0d40e2102dc06"
server_key_version: "encryptionkey@0"
I20260811 11:50:31.466758 10878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:50:31.469543 10903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:31.470963 10878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:50:31.471093 10878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "5c238abf7e7b4e408709b3513fa869e1"
format_stamp: "Formatted at 2026-08-11 11:50:31 on dist-test-slave-bndk"
server_key: "1aad9dba5cc09ea4adc77ceb23a7dc0c"
server_key_iv: "25810179bc856820dce0d40e2102dc06"
server_key_version: "encryptionkey@0"
I20260811 11:50:31.471211 10878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:31.483872 10878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:31.487391 10878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:31.487591 10878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:31.488226 10878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:31.489130 10878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:31.489182 10878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:31.489293 10878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:31.489326 10878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:31.498193 10878 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:40747
I20260811 11:50:31.498210 11071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:40747 every 8 connection(s)
I20260811 11:50:31.498854 10878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:50:31.499714 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 10878
I20260811 11:50:31.499833 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:50:31.500085 3957 external_mini_cluster.cc:1428] Setting key 3087b79076eab48e87ed56c1098df626
Aug 11 11:50:31 dist-test-slave-bndk krb5kdc[10719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449031, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.66@KRBTEST.COM" created.
I20260811 11:50:31.515687 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:31.503351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:52985 (local address 127.3.221.126:43013)
0811 11:50:31.503528 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:31.503534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:31.503554 (+ 20us) server_negotiation.cc:407] Connection header received
0811 11:50:31.504347 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:31.504351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:31.504410 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:31.504484 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:31.505810 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:31.506349 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:31.507515 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:31.507751 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:31.507841 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:31.511185 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:31.511188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:31.511190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:31.511234 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:31.512907 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:31.513632 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:31.513636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:31.513638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:31.513740 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:31.514230 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:31.514234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:31.514236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:31.514503 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:31.514613 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:31.515318 (+ 705us) server_negotiation.cc:299] Negotiation successful
0811 11:50:31.515490 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260811 11:50:31.516558 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:31.502281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:43013 (local address 127.3.221.65:52985)
0811 11:50:31.502858 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:31.502907 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:31.504113 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:31.504687 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:31.504694 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:31.505087 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:31.505652 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:31.505667 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:31.506475 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:31.506477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:31.507383 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:31.507394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:31.507906 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:31.507910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:31.508084 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:31.509001 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:31.509025 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:31.510980 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:31.513050 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:31.513057 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:31.513077 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:31.513503 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:31.513885 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:31.513888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:31.513890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:31.514096 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:31.514626 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:31.514633 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:31.514997 (+ 364us) client_negotiation.cc:769] Sending connection context
0811 11:50:31.515207 (+ 210us) client_negotiation.cc:240] Negotiation successful
0811 11:50:31.515574 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":148,"threads_started":1}
I20260811 11:50:31.520311 11073 heartbeater.cc:347] Connected to a master server at 127.3.221.126:43013
I20260811 11:50:31.520646 11073 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:31.521339 11073 heartbeater.cc:511] Master 127.3.221.126:43013 requested a full tablet report, sending...
I20260811 11:50:31.523244 10779 ts_manager.cc:194] Registered new tserver with Master: 5c238abf7e7b4e408709b3513fa869e1 (127.3.221.65:40747)
I20260811 11:50:31.524804 10779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:52985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
I20260811 11:50:31.532508 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:43013
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:31.647147 11079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:31.647449 11079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:31.647518 11079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:31.651584 11079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:31.651645 11079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:31.651731 11079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:50:31.657174 11079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:43013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:31.658309 11079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:31.659343 11079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:31.669119 11090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:31.669291 11079 server_base.cc:1034] running on GCE node
W20260811 11:50:31.669112 11091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:31.669112 11093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:31.670047 11079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:31.670737 11079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:31.671916 11079 hybrid_clock.cc:648] HybridClock initialized: now 1786449031671872 us; error 51 us; skew 500 ppm
Aug 11 11:50:31 dist-test-slave-bndk krb5kdc[10719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449031, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:31.674844 11079 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:50:31.675734 11079 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:31.675822 11079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:31.676056 11079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:31.678315 11079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "520a2f3020bc41918645db3f75f6ba6a"
format_stamp: "Formatted at 2026-08-11 11:50:31 on dist-test-slave-bndk"
server_key: "1a7e8274dc694fadeedb899aa169ae72"
server_key_iv: "a1ea55ad682408e6aea346ddf8402b88"
server_key_version: "encryptionkey@0"
I20260811 11:50:31.678910 11079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "520a2f3020bc41918645db3f75f6ba6a"
format_stamp: "Formatted at 2026-08-11 11:50:31 on dist-test-slave-bndk"
server_key: "1a7e8274dc694fadeedb899aa169ae72"
server_key_iv: "a1ea55ad682408e6aea346ddf8402b88"
server_key_version: "encryptionkey@0"
I20260811 11:50:31.682906 11079 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:50:31.685554 11101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:31.686960 11079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 11:50:31.687096 11079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "520a2f3020bc41918645db3f75f6ba6a"
format_stamp: "Formatted at 2026-08-11 11:50:31 on dist-test-slave-bndk"
server_key: "1a7e8274dc694fadeedb899aa169ae72"
server_key_iv: "a1ea55ad682408e6aea346ddf8402b88"
server_key_version: "encryptionkey@0"
I20260811 11:50:31.687203 11079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:31.727867 11079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:31.731364 11079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:31.731532 11079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:31.732093 11079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:31.732923 11079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:31.732970 11079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:31.733011 11079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:31.733031 11079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:31.741583 11079 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:40815
I20260811 11:50:31.741622 11264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:40815 every 8 connection(s)
I20260811 11:50:31.742206 11079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:50:31.743196 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 11079
I20260811 11:50:31.743299 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:50:31.743583 3957 external_mini_cluster.cc:1428] Setting key 3054a85ef6436587c4f1a3b08b438458
Aug 11 11:50:31 dist-test-slave-bndk krb5kdc[10719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449031, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.221.67@KRBTEST.COM; defaulting to no policy
I20260811 11:50:31.756376 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:31.744075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:45073 (local address 127.3.221.126:43013)
0811 11:50:31.744232 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:31.744236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:31.745168 (+ 932us) server_negotiation.cc:407] Connection header received
0811 11:50:31.746536 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:31.746539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:31.746592 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:31.746677 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:31.747953 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:31.748492 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:31.749735 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:31.750031 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:31.750132 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:31.752678 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:31.752683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:31.752686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:31.752720 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:31.754263 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:31.754771 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:31.754775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:31.754777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:31.754862 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:31.755183 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:31.755186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:31.755188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:31.755437 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:31.755528 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:31.756058 (+ 530us) server_negotiation.cc:299] Negotiation successful
0811 11:50:31.756193 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 11:50:31.757246 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:31.744537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:43013 (local address 127.3.221.66:45073)
0811 11:50:31.745013 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:31.745059 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:31.746313 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:31.746888 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:31.746895 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:31.747271 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:31.747785 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:31.747799 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:31.748621 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:31.748623 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:31.749598 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:31.749628 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:31.750177 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:31.750180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:31.750353 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:31.750923 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:31.750940 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:31.752483 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:31.754388 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:31.754393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:31.754407 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:31.754674 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:31.754948 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:31.754951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:31.754952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:31.755094 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:31.755539 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:31.755546 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:31.755815 (+ 269us) client_negotiation.cc:769] Sending connection context
0811 11:50:31.755997 (+ 182us) client_negotiation.cc:240] Negotiation successful
0811 11:50:31.756342 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.67@KRBTEST.COM" created.
I20260811 11:50:31.758476 11265 heartbeater.cc:347] Connected to a master server at 127.3.221.126:43013
I20260811 11:50:31.758740 11265 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:31.759279 11265 heartbeater.cc:511] Master 127.3.221.126:43013 requested a full tablet report, sending...
I20260811 11:50:31.760452 10779 ts_manager.cc:194] Registered new tserver with Master: 520a2f3020bc41918645db3f75f6ba6a (127.3.221.66:40815)
I20260811 11:50:31.761200 10779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:45073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
I20260811 11:50:31.774606 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:43013
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:31.891122 11273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:31.891489 11273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:31.891568 11273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:31.895756 11273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:31.895840 11273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:31.895926 11273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:50:31.901440 11273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:43013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:31.902567 11273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:31.903607 11273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:31.913334 11287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:31.913333 11286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:31.913367 11290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:31.913539 11273 server_base.cc:1034] running on GCE node
I20260811 11:50:31.913966 11273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:31.914583 11273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:31.915738 11273 hybrid_clock.cc:648] HybridClock initialized: now 1786449031915724 us; error 34 us; skew 500 ppm
Aug 11 11:50:31 dist-test-slave-bndk krb5kdc[10719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449031, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:31.918817 11273 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:50:31.919598 11273 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:31.919664 11273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:31.919837 11273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:31.921993 11273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "4b2c8801b87840d68b3ce748df863920"
format_stamp: "Formatted at 2026-08-11 11:50:31 on dist-test-slave-bndk"
server_key: "99a6d9387d8afb3b76d2ea0f921f349e"
server_key_iv: "40caa1ca960c3fa6b0f40874136b029e"
server_key_version: "encryptionkey@0"
I20260811 11:50:31.922533 11273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "4b2c8801b87840d68b3ce748df863920"
format_stamp: "Formatted at 2026-08-11 11:50:31 on dist-test-slave-bndk"
server_key: "99a6d9387d8afb3b76d2ea0f921f349e"
server_key_iv: "40caa1ca960c3fa6b0f40874136b029e"
server_key_version: "encryptionkey@0"
I20260811 11:50:31.926313 11273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:50:31.928876 11300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:31.930192 11273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 11:50:31.930298 11273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "4b2c8801b87840d68b3ce748df863920"
format_stamp: "Formatted at 2026-08-11 11:50:31 on dist-test-slave-bndk"
server_key: "99a6d9387d8afb3b76d2ea0f921f349e"
server_key_iv: "40caa1ca960c3fa6b0f40874136b029e"
server_key_version: "encryptionkey@0"
I20260811 11:50:31.930392 11273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:31.946786 11273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:31.949787 11273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:31.949950 11273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:31.950505 11273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:31.951409 11273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:31.951476 11273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:31.951530 11273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:31.951560 11273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:31.960759 11273 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:38209
I20260811 11:50:31.960779 11493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:38209 every 8 connection(s)
I20260811 11:50:31.961450 11273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:50:31.962549 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 11273
I20260811 11:50:31.962666 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:50:31.962998 3957 external_mini_cluster.cc:1428] Setting key b38cf31257a0d1115cf8c025b8351eb4
Aug 11 11:50:31 dist-test-slave-bndk krb5kdc[10719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449031, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:31.975121 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:31.963388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:47527 (local address 127.3.221.126:43013)
0811 11:50:31.963575 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:31.963579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:31.964515 (+ 936us) server_negotiation.cc:407] Connection header received
0811 11:50:31.965970 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:31.965973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:31.966023 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:31.966092 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:31.967262 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:31.967815 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:31.969017 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:31.969283 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:31.969377 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:31.971900 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:31.971903 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:31.971905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:31.971932 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:31.973262 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:31.973737 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:31.973740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:31.973741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:31.973816 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:31.974128 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:31.974130 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:31.974131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:31.974319 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:31.974394 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:31.974858 (+ 464us) server_negotiation.cc:299] Negotiation successful
0811 11:50:31.974969 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 11:50:31.976099 11500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:31.963828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:43013 (local address 127.3.221.67:47527)
0811 11:50:31.964384 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:31.964433 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:31.965759 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:31.966232 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:31.966238 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:31.966610 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:31.967098 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:31.967112 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:31.967947 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:31.967949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:31.968864 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:31.968875 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:31.969439 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:31.969443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:31.969612 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:31.970180 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:31.970197 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:31.971704 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:31.973368 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:31.973373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:31.973387 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:31.973657 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:31.973890 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:31.973892 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:31.973893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:31.974058 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:31.974397 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:31.974403 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:31.974659 (+ 256us) client_negotiation.cc:769] Sending connection context
0811 11:50:31.974819 (+ 160us) client_negotiation.cc:240] Negotiation successful
0811 11:50:31.975093 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":144,"threads_started":1}
I20260811 11:50:31.977437 11497 heartbeater.cc:347] Connected to a master server at 127.3.221.126:43013
I20260811 11:50:31.977718 11497 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:31.978188 11497 heartbeater.cc:511] Master 127.3.221.126:43013 requested a full tablet report, sending...
I20260811 11:50:31.979285 10776 ts_manager.cc:194] Registered new tserver with Master: 4b2c8801b87840d68b3ce748df863920 (127.3.221.67:38209)
I20260811 11:50:31.979868 10776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:47527
I20260811 11:50:31.989329 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:50:31.998411 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:31.991352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44420 (local address 127.3.221.126:43013)
0811 11:50:31.991525 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:31.991530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:31.991782 (+ 252us) server_negotiation.cc:407] Connection header received
0811 11:50:31.991916 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:31.991918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:31.991967 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:31.992027 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:31.992778 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:31.993360 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:31.994194 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:31.994438 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:31.994519 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:31.995800 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:31.995803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:31.995805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:31.995832 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:31.996975 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:31.997400 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:31.997402 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:31.997404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:31.997483 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:31.997674 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:31.997675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:31.997677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:31.997891 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:31.997993 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:31.998144 (+ 151us) server_negotiation.cc:299] Negotiation successful
0811 11:50:31.998243 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 11:50:32.001600 10776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:50:32.004585 10776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:50:32.019479 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:32.014122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49502 (local address 127.3.221.65:40747)
0811 11:50:32.014896 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:32.014902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:32.014924 (+ 22us) server_negotiation.cc:407] Connection header received
0811 11:50:32.014993 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:32.014998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:32.015166 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:32.015471 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:32.015625 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:32.016385 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:32.017954 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:32.018799 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:32.018916 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:32.019083 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:32.019252 (+ 169us) server_negotiation.cc:299] Negotiation successful
0811 11:50:32.019337 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":96,"threads_started":1}
I20260811 11:50:32.019515 11514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:32.013205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:40747 (local address 127.0.0.1:49502)
0811 11:50:32.014130 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:32.014186 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:32.014327 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:32.015317 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:32.015320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:32.015349 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:32.015493 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:32.015501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:32.016510 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:32.016512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:32.017832 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:32.017842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:32.018932 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:32.018936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:32.019052 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:32.019100 (+ 48us) client_negotiation.cc:769] Sending connection context
0811 11:50:32.019215 (+ 115us) client_negotiation.cc:240] Negotiation successful
0811 11:50:32.019356 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":90,"spinlock_wait_cycles":13952,"thread_start_us":111,"threads_started":1}
I20260811 11:50:32.020021 11515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:32.013471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:40815 (local address 127.0.0.1:34278)
0811 11:50:32.014506 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:32.014518 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:32.014590 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:32.015321 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:32.015323 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:32.015348 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:32.015648 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:32.015655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:32.017096 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:32.017099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:32.018285 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:32.018293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:32.019290 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:32.019293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:32.019380 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:32.019400 (+ 20us) client_negotiation.cc:769] Sending connection context
0811 11:50:32.019469 (+ 69us) client_negotiation.cc:240] Negotiation successful
0811 11:50:32.019527 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976,"mutex_wait_us":45,"spinlock_wait_cycles":1792,"thread_start_us":108,"threads_started":1}
I20260811 11:50:32.020033 11521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:32.014501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.3.221.66:40815)
0811 11:50:32.014826 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:32.014835 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:32.014854 (+ 19us) server_negotiation.cc:407] Connection header received
0811 11:50:32.014915 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:32.014918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:32.015064 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:32.015209 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:32.015770 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:32.016833 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:32.018403 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:32.018981 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:32.019080 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:32.019230 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:32.019535 (+ 305us) server_negotiation.cc:299] Negotiation successful
0811 11:50:32.019888 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":112,"threads_started":1}
I20260811 11:50:32.020375 11516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:32.013412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48454 (local address 127.3.221.67:38209)
0811 11:50:32.014620 (+ 1208us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:32.014632 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:32.014657 (+ 25us) server_negotiation.cc:407] Connection header received
0811 11:50:32.014735 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:32.014741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:32.014958 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:32.015124 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:32.015440 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:32.016514 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:32.018077 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:32.018817 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:32.018916 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:32.019074 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:32.020111 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0811 11:50:32.020214 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1104,"thread_start_us":163,"threads_started":1}
I20260811 11:50:32.020040 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:32.013191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:38209 (local address 127.0.0.1:48454)
0811 11:50:32.014130 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:32.014186 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:32.014327 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:32.015133 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:32.015140 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:32.015164 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:32.015312 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:32.015320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:32.016649 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:32.016652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:32.017953 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:32.017965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:32.019704 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:32.019708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:32.019787 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:32.019808 (+ 21us) client_negotiation.cc:769] Sending connection context
0811 11:50:32.019872 (+ 64us) client_negotiation.cc:240] Negotiation successful
0811 11:50:32.019933 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":36,"spinlock_wait_cycles":14592,"thread_start_us":105,"threads_started":1}
I20260811 11:50:32.022734 10973 tablet_service.cc:1467] Processing CreateTablet for tablet 7d31e26448f94c74acd9faa9e3a2cc4e (DEFAULT_TABLE table=test-table [id=f719b052214449999da4336897ce28dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:32.022981 11172 tablet_service.cc:1467] Processing CreateTablet for tablet 7d31e26448f94c74acd9faa9e3a2cc4e (DEFAULT_TABLE table=test-table [id=f719b052214449999da4336897ce28dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:32.023764 10973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d31e26448f94c74acd9faa9e3a2cc4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:32.023952 11172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d31e26448f94c74acd9faa9e3a2cc4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:32.028251 11376 tablet_service.cc:1467] Processing CreateTablet for tablet 7d31e26448f94c74acd9faa9e3a2cc4e (DEFAULT_TABLE table=test-table [id=f719b052214449999da4336897ce28dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:32.029582 11376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d31e26448f94c74acd9faa9e3a2cc4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:32.030512 11524 tablet_bootstrap.cc:492] T 7d31e26448f94c74acd9faa9e3a2cc4e P 5c238abf7e7b4e408709b3513fa869e1: Bootstrap starting.
I20260811 11:50:32.032042 11525 tablet_bootstrap.cc:492] T 7d31e26448f94c74acd9faa9e3a2cc4e P 520a2f3020bc41918645db3f75f6ba6a: Bootstrap starting.
I20260811 11:50:32.033108 11524 tablet_bootstrap.cc:654] T 7d31e26448f94c74acd9faa9e3a2cc4e P 5c238abf7e7b4e408709b3513fa869e1: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:32.034155 11524 log.cc:824] T 7d31e26448f94c74acd9faa9e3a2cc4e P 5c238abf7e7b4e408709b3513fa869e1: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:32.034274 11525 tablet_bootstrap.cc:654] T 7d31e26448f94c74acd9faa9e3a2cc4e P 520a2f3020bc41918645db3f75f6ba6a: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:32.035373 11525 log.cc:824] T 7d31e26448f94c74acd9faa9e3a2cc4e P 520a2f3020bc41918645db3f75f6ba6a: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:32.036629 11524 tablet_bootstrap.cc:492] T 7d31e26448f94c74acd9faa9e3a2cc4e P 5c238abf7e7b4e408709b3513fa869e1: No bootstrap required, opened a new log
I20260811 11:50:32.036859 11524 ts_tablet_manager.cc:1397] T 7d31e26448f94c74acd9faa9e3a2cc4e P 5c238abf7e7b4e408709b3513fa869e1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260811 11:50:32.037267 11525 tablet_bootstrap.cc:492] T 7d31e26448f94c74acd9faa9e3a2cc4e P 520a2f3020bc41918645db3f75f6ba6a: No bootstrap required, opened a new log
I20260811 11:50:32.037477 11525 ts_tablet_manager.cc:1397] T 7d31e26448f94c74acd9faa9e3a2cc4e P 520a2f3020bc41918645db3f75f6ba6a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 11:50:32.038010 11530 tablet_bootstrap.cc:492] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920: Bootstrap starting.
I20260811 11:50:32.040372 11530 tablet_bootstrap.cc:654] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:32.040802 11524 raft_consensus.cc:348] T 7d31e26448f94c74acd9faa9e3a2cc4e P 5c238abf7e7b4e408709b3513fa869e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2c8801b87840d68b3ce748df863920" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 38209 } } peers { permanent_uuid: "5c238abf7e7b4e408709b3513fa869e1" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40747 } } peers { permanent_uuid: "520a2f3020bc41918645db3f75f6ba6a" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40815 } }
I20260811 11:50:32.041036 11524 raft_consensus.cc:374] T 7d31e26448f94c74acd9faa9e3a2cc4e P 5c238abf7e7b4e408709b3513fa869e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:32.041090 11524 raft_consensus.cc:729] T 7d31e26448f94c74acd9faa9e3a2cc4e P 5c238abf7e7b4e408709b3513fa869e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c238abf7e7b4e408709b3513fa869e1, State: Initialized, Role: FOLLOWER
I20260811 11:50:32.041257 11525 raft_consensus.cc:348] T 7d31e26448f94c74acd9faa9e3a2cc4e P 520a2f3020bc41918645db3f75f6ba6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2c8801b87840d68b3ce748df863920" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 38209 } } peers { permanent_uuid: "5c238abf7e7b4e408709b3513fa869e1" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40747 } } peers { permanent_uuid: "520a2f3020bc41918645db3f75f6ba6a" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40815 } }
I20260811 11:50:32.041632 11525 raft_consensus.cc:374] T 7d31e26448f94c74acd9faa9e3a2cc4e P 520a2f3020bc41918645db3f75f6ba6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:32.041580 11524 consensus_queue.cc:260] T 7d31e26448f94c74acd9faa9e3a2cc4e P 5c238abf7e7b4e408709b3513fa869e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2c8801b87840d68b3ce748df863920" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 38209 } } peers { permanent_uuid: "5c238abf7e7b4e408709b3513fa869e1" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40747 } } peers { permanent_uuid: "520a2f3020bc41918645db3f75f6ba6a" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40815 } }
I20260811 11:50:32.041705 11525 raft_consensus.cc:729] T 7d31e26448f94c74acd9faa9e3a2cc4e P 520a2f3020bc41918645db3f75f6ba6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 520a2f3020bc41918645db3f75f6ba6a, State: Initialized, Role: FOLLOWER
I20260811 11:50:32.041697 11530 log.cc:824] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:32.042191 11525 consensus_queue.cc:260] T 7d31e26448f94c74acd9faa9e3a2cc4e P 520a2f3020bc41918645db3f75f6ba6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2c8801b87840d68b3ce748df863920" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 38209 } } peers { permanent_uuid: "5c238abf7e7b4e408709b3513fa869e1" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40747 } } peers { permanent_uuid: "520a2f3020bc41918645db3f75f6ba6a" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40815 } }
I20260811 11:50:32.042344 11073 heartbeater.cc:503] Master 127.3.221.126:43013 was elected leader, sending a full tablet report...
I20260811 11:50:32.042652 11524 ts_tablet_manager.cc:1428] T 7d31e26448f94c74acd9faa9e3a2cc4e P 5c238abf7e7b4e408709b3513fa869e1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 11:50:32.042914 11265 heartbeater.cc:503] Master 127.3.221.126:43013 was elected leader, sending a full tablet report...
I20260811 11:50:32.043148 11525 ts_tablet_manager.cc:1428] T 7d31e26448f94c74acd9faa9e3a2cc4e P 520a2f3020bc41918645db3f75f6ba6a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260811 11:50:32.045277 11530 tablet_bootstrap.cc:492] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920: No bootstrap required, opened a new log
I20260811 11:50:32.045562 11530 ts_tablet_manager.cc:1397] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260811 11:50:32.049054 11530 raft_consensus.cc:348] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2c8801b87840d68b3ce748df863920" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 38209 } } peers { permanent_uuid: "5c238abf7e7b4e408709b3513fa869e1" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40747 } } peers { permanent_uuid: "520a2f3020bc41918645db3f75f6ba6a" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40815 } }
I20260811 11:50:32.049336 11530 raft_consensus.cc:374] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:32.049398 11530 raft_consensus.cc:729] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b2c8801b87840d68b3ce748df863920, State: Initialized, Role: FOLLOWER
I20260811 11:50:32.049837 11530 consensus_queue.cc:260] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2c8801b87840d68b3ce748df863920" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 38209 } } peers { permanent_uuid: "5c238abf7e7b4e408709b3513fa869e1" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40747 } } peers { permanent_uuid: "520a2f3020bc41918645db3f75f6ba6a" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40815 } }
I20260811 11:50:32.050524 11497 heartbeater.cc:503] Master 127.3.221.126:43013 was elected leader, sending a full tablet report...
I20260811 11:50:32.050840 11530 ts_tablet_manager.cc:1428] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260811 11:50:32.212414 11498 tablet.cc:2369] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:50:32.237727 11540 raft_consensus.cc:482] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:50:32.237984 11540 raft_consensus.cc:504] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2c8801b87840d68b3ce748df863920" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 38209 } } peers { permanent_uuid: "5c238abf7e7b4e408709b3513fa869e1" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40747 } } peers { permanent_uuid: "520a2f3020bc41918645db3f75f6ba6a" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40815 } }
I20260811 11:50:32.239176 11540 leader_election.cc:288] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c238abf7e7b4e408709b3513fa869e1 (127.3.221.65:40747), 520a2f3020bc41918645db3f75f6ba6a (127.3.221.66:40815)
I20260811 11:50:32.242899 11521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:32.239608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:53899 (local address 127.3.221.66:40815)
0811 11:50:32.239722 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:32.239726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:32.239745 (+ 19us) server_negotiation.cc:407] Connection header received
0811 11:50:32.239875 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:32.239878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:32.239933 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:32.240011 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:32.240299 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:32.240807 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:32.241995 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:32.242396 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:32.242502 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:32.242561 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:32.242713 (+ 152us) server_negotiation.cc:299] Negotiation successful
0811 11:50:32.242791 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260811 11:50:32.242902 11500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:32.239496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:40815 (local address 127.3.221.67:53899)
0811 11:50:32.239627 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:32.239643 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:32.239750 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:32.240035 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:32.240038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:32.240056 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:32.240185 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:32.240191 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:32.240913 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:32.240916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:32.241870 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:32.241881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:32.242502 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:32.242506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:32.242606 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:32.242623 (+ 17us) client_negotiation.cc:769] Sending connection context
0811 11:50:32.242689 (+ 66us) client_negotiation.cc:240] Negotiation successful
0811 11:50:32.242758 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":20,"spinlock_wait_cycles":2432}
I20260811 11:50:32.243052 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:32.239609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:35597 (local address 127.3.221.65:40747)
0811 11:50:32.239725 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:32.239729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:32.240086 (+ 357us) server_negotiation.cc:407] Connection header received
0811 11:50:32.240204 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:32.240206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:32.240247 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:32.240308 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:32.240556 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:32.241046 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:32.242121 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:32.242522 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:32.242626 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:32.242677 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:32.242847 (+ 170us) server_negotiation.cc:299] Negotiation successful
0811 11:50:32.242918 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260811 11:50:32.243291 11266 tablet.cc:2369] T 7d31e26448f94c74acd9faa9e3a2cc4e P 520a2f3020bc41918645db3f75f6ba6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:50:32.243435 11542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:32.239498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:40747 (local address 127.3.221.67:35597)
0811 11:50:32.239993 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:32.240004 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:32.240105 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:32.240332 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:32.240334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:32.240350 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:32.240458 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:32.240464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:32.241154 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:32.241157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:32.242000 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:32.242009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:32.242630 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:32.242633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:32.242735 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:32.242754 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 11:50:32.242828 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 11:50:32.242894 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":164,"spinlock_wait_cycles":21248,"thread_start_us":129,"threads_started":1}
I20260811 11:50:32.243588 11202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d31e26448f94c74acd9faa9e3a2cc4e" candidate_uuid: "4b2c8801b87840d68b3ce748df863920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "520a2f3020bc41918645db3f75f6ba6a" is_pre_election: true
I20260811 11:50:32.243908 11202 raft_consensus.cc:2454] T 7d31e26448f94c74acd9faa9e3a2cc4e P 520a2f3020bc41918645db3f75f6ba6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b2c8801b87840d68b3ce748df863920 in term 0.
I20260811 11:50:32.244050 10997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d31e26448f94c74acd9faa9e3a2cc4e" candidate_uuid: "4b2c8801b87840d68b3ce748df863920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c238abf7e7b4e408709b3513fa869e1" is_pre_election: true
I20260811 11:50:32.244325 10997 raft_consensus.cc:2454] T 7d31e26448f94c74acd9faa9e3a2cc4e P 5c238abf7e7b4e408709b3513fa869e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b2c8801b87840d68b3ce748df863920 in term 0.
I20260811 11:50:32.244400 11305 leader_election.cc:302] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b2c8801b87840d68b3ce748df863920, 520a2f3020bc41918645db3f75f6ba6a; no voters:
I20260811 11:50:32.244664 11540 raft_consensus.cc:2781] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:50:32.244740 11540 raft_consensus.cc:482] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:50:32.244781 11540 raft_consensus.cc:3037] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:32.245918 11540 raft_consensus.cc:504] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2c8801b87840d68b3ce748df863920" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 38209 } } peers { permanent_uuid: "5c238abf7e7b4e408709b3513fa869e1" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40747 } } peers { permanent_uuid: "520a2f3020bc41918645db3f75f6ba6a" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40815 } }
I20260811 11:50:32.246254 11540 leader_election.cc:288] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920 [CANDIDATE]: Term 1 election: Requested vote from peers 5c238abf7e7b4e408709b3513fa869e1 (127.3.221.65:40747), 520a2f3020bc41918645db3f75f6ba6a (127.3.221.66:40815)
I20260811 11:50:32.246628 10997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d31e26448f94c74acd9faa9e3a2cc4e" candidate_uuid: "4b2c8801b87840d68b3ce748df863920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c238abf7e7b4e408709b3513fa869e1"
I20260811 11:50:32.246627 11202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d31e26448f94c74acd9faa9e3a2cc4e" candidate_uuid: "4b2c8801b87840d68b3ce748df863920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "520a2f3020bc41918645db3f75f6ba6a"
I20260811 11:50:32.246754 10997 raft_consensus.cc:3037] T 7d31e26448f94c74acd9faa9e3a2cc4e P 5c238abf7e7b4e408709b3513fa869e1 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:32.246768 11202 raft_consensus.cc:3037] T 7d31e26448f94c74acd9faa9e3a2cc4e P 520a2f3020bc41918645db3f75f6ba6a [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:32.248039 11202 raft_consensus.cc:2454] T 7d31e26448f94c74acd9faa9e3a2cc4e P 520a2f3020bc41918645db3f75f6ba6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b2c8801b87840d68b3ce748df863920 in term 1.
I20260811 11:50:32.248039 10997 raft_consensus.cc:2454] T 7d31e26448f94c74acd9faa9e3a2cc4e P 5c238abf7e7b4e408709b3513fa869e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b2c8801b87840d68b3ce748df863920 in term 1.
I20260811 11:50:32.248339 11305 leader_election.cc:302] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b2c8801b87840d68b3ce748df863920, 520a2f3020bc41918645db3f75f6ba6a; no voters:
I20260811 11:50:32.248512 11540 raft_consensus.cc:2781] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:32.248821 11540 raft_consensus.cc:686] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920 [term 1 LEADER]: Becoming Leader. State: Replica: 4b2c8801b87840d68b3ce748df863920, State: Running, Role: LEADER
I20260811 11:50:32.249190 11540 consensus_queue.cc:237] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2c8801b87840d68b3ce748df863920" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 38209 } } peers { permanent_uuid: "5c238abf7e7b4e408709b3513fa869e1" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40747 } } peers { permanent_uuid: "520a2f3020bc41918645db3f75f6ba6a" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40815 } }
W20260811 11:50:32.250160 11074 tablet.cc:2369] T 7d31e26448f94c74acd9faa9e3a2cc4e P 5c238abf7e7b4e408709b3513fa869e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:50:32.252667 10779 catalog_manager.cc:5495] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b2c8801b87840d68b3ce748df863920 (127.3.221.67). New cstate: current_term: 1 leader_uuid: "4b2c8801b87840d68b3ce748df863920" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2c8801b87840d68b3ce748df863920" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 38209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c238abf7e7b4e408709b3513fa869e1" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "520a2f3020bc41918645db3f75f6ba6a" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40815 } health_report { overall_health: UNKNOWN } } }
I20260811 11:50:32.314890 11516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:32.311310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48464 (local address 127.3.221.67:38209)
0811 11:50:32.311439 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:32.311443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:32.311457 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:50:32.311540 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:32.311542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:32.311600 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:32.311701 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:32.311925 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:32.312433 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:32.313362 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:32.313585 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:32.313678 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:32.314045 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:32.314530 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:32.314634 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:32.314713 (+ 79us) server_negotiation.cc:299] Negotiation successful
0811 11:50:32.314771 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 11:50:32.321027 11202 raft_consensus.cc:1261] T 7d31e26448f94c74acd9faa9e3a2cc4e P 520a2f3020bc41918645db3f75f6ba6a [term 1 FOLLOWER]: Refusing update from remote peer 4b2c8801b87840d68b3ce748df863920: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:32.321027 10997 raft_consensus.cc:1261] T 7d31e26448f94c74acd9faa9e3a2cc4e P 5c238abf7e7b4e408709b3513fa869e1 [term 1 FOLLOWER]: Refusing update from remote peer 4b2c8801b87840d68b3ce748df863920: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:32.321781 11545 consensus_queue.cc:1035] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920 [LEADER]: Connected to new peer: Peer: permanent_uuid: "520a2f3020bc41918645db3f75f6ba6a" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:32.321996 11540 consensus_queue.cc:1035] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c238abf7e7b4e408709b3513fa869e1" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:32.330308 11554 mvcc.cc:205] Tried to move back new op lower bound from 7317295236379250688 to 7317295236095447040. Current Snapshot: MvccSnapshot[applied={T|T < 7317295236379250688}]
I20260811 11:50:32.330904 11552 mvcc.cc:205] Tried to move back new op lower bound from 7317295236379250688 to 7317295236095447040. Current Snapshot: MvccSnapshot[applied={T|T < 7317295236379250688}]
I20260811 11:50:32.331071 11553 mvcc.cc:205] Tried to move back new op lower bound from 7317295236379250688 to 7317295236095447040. Current Snapshot: MvccSnapshot[applied={T|T < 7317295236379250688}]
I20260811 11:50:32.341457 10775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:32.341675 10775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:32.344501 10775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ced5041a0e74ccda312f6fee6249ee5: Sending DeleteTablet for 3 replicas of tablet 7d31e26448f94c74acd9faa9e3a2cc4e
I20260811 11:50:32.345368 11376 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d31e26448f94c74acd9faa9e3a2cc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:32 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:48454
I20260811 11:50:32.345418 10973 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d31e26448f94c74acd9faa9e3a2cc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:32 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:49502
I20260811 11:50:32.345961 11172 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d31e26448f94c74acd9faa9e3a2cc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:32 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:34278
I20260811 11:50:32.346429 11570 tablet_replica.cc:330] T 7d31e26448f94c74acd9faa9e3a2cc4e P 520a2f3020bc41918645db3f75f6ba6a: stopping tablet replica
I20260811 11:50:32.346624 11565 tablet_replica.cc:330] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920: stopping tablet replica
I20260811 11:50:32.346805 11570 raft_consensus.cc:2229] T 7d31e26448f94c74acd9faa9e3a2cc4e P 520a2f3020bc41918645db3f75f6ba6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:32.347182 11570 raft_consensus.cc:2258] T 7d31e26448f94c74acd9faa9e3a2cc4e P 520a2f3020bc41918645db3f75f6ba6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:32.347400 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 10878
I20260811 11:50:32.352113 11570 ts_tablet_manager.cc:1905] T 7d31e26448f94c74acd9faa9e3a2cc4e P 520a2f3020bc41918645db3f75f6ba6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:32.355674 11570 ts_tablet_manager.cc:1918] T 7d31e26448f94c74acd9faa9e3a2cc4e P 520a2f3020bc41918645db3f75f6ba6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:50:32.355818 11570 log.cc:1196] T 7d31e26448f94c74acd9faa9e3a2cc4e P 520a2f3020bc41918645db3f75f6ba6a: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/7d31e26448f94c74acd9faa9e3a2cc4e
I20260811 11:50:32.356201 11570 ts_tablet_manager.cc:1939] T 7d31e26448f94c74acd9faa9e3a2cc4e P 520a2f3020bc41918645db3f75f6ba6a: Deleting consensus metadata
I20260811 11:50:32.358031 10760 catalog_manager.cc:4841] TS 520a2f3020bc41918645db3f75f6ba6a (127.3.221.66:40815): tablet 7d31e26448f94c74acd9faa9e3a2cc4e (table test-table [id=f719b052214449999da4336897ce28dd]) successfully deleted
I20260811 11:50:32.361783 11565 raft_consensus.cc:2229] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:50:32.362301 11565 raft_consensus.cc:2258] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:32.364049 11565 ts_tablet_manager.cc:1905] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:32.366972 11565 ts_tablet_manager.cc:1918] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:50:32.367093 11565 log.cc:1196] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/7d31e26448f94c74acd9faa9e3a2cc4e
I20260811 11:50:32.367434 11565 ts_tablet_manager.cc:1939] T 7d31e26448f94c74acd9faa9e3a2cc4e P 4b2c8801b87840d68b3ce748df863920: Deleting consensus metadata
W20260811 11:50:32.367528 10758 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.221.65:40747 (error 108)
I20260811 11:50:32.367725 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 11079
I20260811 11:50:32.376999 10761 catalog_manager.cc:4841] TS 4b2c8801b87840d68b3ce748df863920 (127.3.221.67:38209): tablet 7d31e26448f94c74acd9faa9e3a2cc4e (table test-table [id=f719b052214449999da4336897ce28dd]) successfully deleted
I20260811 11:50:32.377779 11515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:32.377403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:40747 (local address 127.0.0.1:49512)
0811 11:50:32.377579 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:32.377677 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.65:40747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260811 11:50:32.378057 10758 catalog_manager.cc:4584] TS 5c238abf7e7b4e408709b3513fa869e1 (127.3.221.65:40747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d31e26448f94c74acd9faa9e3a2cc4e: Network error: Client connection negotiation failed: client connection to 127.3.221.65:40747: connect: Connection refused (error 111)
I20260811 11:50:32.383134 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 11273
I20260811 11:50:32.397099 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 10739
2026-08-11T11:50:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4112 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-11T11:50:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:50:32Z Disabled control of system clock
I20260811 11:50:32.438220 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:43673
--webserver_interface=127.3.221.126
--webserver_port=0
--builtin_ntp_servers=127.3.221.84:46181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:43673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260811 11:50:32.554384 11582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:32.554668 11582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:32.554706 11582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:32.558847 11582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:50:32.558925 11582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:32.558939 11582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:32.558955 11582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:50:32.558969 11582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:50:32.563984 11582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:46181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:43673
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:43673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:32.565164 11582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:32.566628 11582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:32.575474 11591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:32.575507 11587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:32.575659 11582 server_base.cc:1034] running on GCE node
W20260811 11:50:32.575577 11588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:32.576363 11582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:32.577646 11582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:32.578845 11582 hybrid_clock.cc:648] HybridClock initialized: now 1786449032578818 us; error 46 us; skew 500 ppm
I20260811 11:50:32.580935 11582 webserver.cc:460] Webserver started at http://127.3.221.126:34089/ using document root <none> and password file <none>
I20260811 11:50:32.581701 11582 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:32.581799 11582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:32.582031 11582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:32.584208 11582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "10677dc2e4a24767bcd3fac9fbced12d"
format_stamp: "Formatted at 2026-08-11 11:50:32 on dist-test-slave-bndk"
server_key: "554608969b3342cf7a130e36b75ae25d"
server_key_iv: "584a2837943e4dd5eb42dfe9f2d6677a"
server_key_version: "encryptionkey@0"
I20260811 11:50:32.584806 11582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "10677dc2e4a24767bcd3fac9fbced12d"
format_stamp: "Formatted at 2026-08-11 11:50:32 on dist-test-slave-bndk"
server_key: "554608969b3342cf7a130e36b75ae25d"
server_key_iv: "584a2837943e4dd5eb42dfe9f2d6677a"
server_key_version: "encryptionkey@0"
I20260811 11:50:32.588722 11582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 11:50:32.591531 11600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:32.592885 11582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 11:50:32.593015 11582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "10677dc2e4a24767bcd3fac9fbced12d"
format_stamp: "Formatted at 2026-08-11 11:50:32 on dist-test-slave-bndk"
server_key: "554608969b3342cf7a130e36b75ae25d"
server_key_iv: "584a2837943e4dd5eb42dfe9f2d6677a"
server_key_version: "encryptionkey@0"
I20260811 11:50:32.593128 11582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:32.597920 11582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:32.598562 11582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:32.598727 11582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:32.605885 11582 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:43673
I20260811 11:50:32.605888 11692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:43673 every 8 connection(s)
I20260811 11:50:32.606904 11582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:50:32.610495 11696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:32.614600 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 11582
I20260811 11:50:32.614725 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:50:32.615036 3957 external_mini_cluster.cc:1428] Setting key 7f6c22bcb11968e55039241c9d70c877
I20260811 11:50:32.617156 11696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d: Bootstrap starting.
I20260811 11:50:32.619578 11696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:32.620419 11696 log.cc:824] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:32.621203 11701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:32.616627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44062 (local address 127.3.221.126:43673)
0811 11:50:32.617426 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:32.617445 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:32.617487 (+ 42us) server_negotiation.cc:407] Connection header received
0811 11:50:32.618191 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:32.618209 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:32.618586 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:32.618926 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:32.619741 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:32.619745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:32.619758 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:50:32.619777 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:32.619836 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:50:32.619853 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:32.619952 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:32.620082 (+ 130us) server_negotiation.cc:299] Negotiation successful
0811 11:50:32.620340 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":153,"threads_started":1}
I20260811 11:50:32.622718 11696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d: No bootstrap required, opened a new log
I20260811 11:50:32.626494 11696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10677dc2e4a24767bcd3fac9fbced12d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 43673 } }
I20260811 11:50:32.626767 11696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:32.626848 11696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10677dc2e4a24767bcd3fac9fbced12d, State: Initialized, Role: FOLLOWER
I20260811 11:50:32.627305 11696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10677dc2e4a24767bcd3fac9fbced12d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 43673 } }
I20260811 11:50:32.627449 11696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:50:32.627519 11696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:50:32.627622 11696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:32.628608 11696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10677dc2e4a24767bcd3fac9fbced12d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 43673 } }
I20260811 11:50:32.628968 11696 leader_election.cc:302] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10677dc2e4a24767bcd3fac9fbced12d; no voters:
I20260811 11:50:32.629328 11696 leader_election.cc:288] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:50:32.629448 11705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:32.629798 11705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d [term 1 LEADER]: Becoming Leader. State: Replica: 10677dc2e4a24767bcd3fac9fbced12d, State: Running, Role: LEADER
I20260811 11:50:32.630149 11705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10677dc2e4a24767bcd3fac9fbced12d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 43673 } }
I20260811 11:50:32.630626 11696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:50:32.631763 11708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10677dc2e4a24767bcd3fac9fbced12d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10677dc2e4a24767bcd3fac9fbced12d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 43673 } } }
I20260811 11:50:32.631886 11708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:32.632176 11706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10677dc2e4a24767bcd3fac9fbced12d. Latest consensus state: current_term: 1 leader_uuid: "10677dc2e4a24767bcd3fac9fbced12d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10677dc2e4a24767bcd3fac9fbced12d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 43673 } } }
I20260811 11:50:32.632279 11706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:32.632395 11723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:50:32.635551 11723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:50:32.641469 11723 catalog_manager.cc:1347] Generated new cluster ID: d826017bfef143b095f10917d0fbdded
I20260811 11:50:32.641553 11723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:50:32.653878 11723 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:50:32.654264 11723 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:50:32.664259 11723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d: Generated new TSK 0
I20260811 11:50:32.665164 11723 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:50:32.676813 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:43673
--builtin_ntp_servers=127.3.221.84:46181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260811 11:50:32.795215 11735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:32.795524 11735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:32.795567 11735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:32.799882 11735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:32.799962 11735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:32.800022 11735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:50:32.804983 11735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:46181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:43673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:32.806206 11735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:32.807674 11735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:32.817868 11745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:32.817889 11750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:32.817934 11735 server_base.cc:1034] running on GCE node
W20260811 11:50:32.817898 11748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:32.818580 11735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:32.819267 11735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:32.820465 11735 hybrid_clock.cc:648] HybridClock initialized: now 1786449032820430 us; error 47 us; skew 500 ppm
I20260811 11:50:32.822489 11735 webserver.cc:460] Webserver started at http://127.3.221.65:40545/ using document root <none> and password file <none>
I20260811 11:50:32.823107 11735 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:32.823182 11735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:32.823417 11735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:32.825704 11735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "b82d4af0068c4b4d8bc9c5a6583398a2"
format_stamp: "Formatted at 2026-08-11 11:50:32 on dist-test-slave-bndk"
server_key: "22b60e12ba9c1162f51e87232a5f6b1c"
server_key_iv: "2fbd107abfcaf382cd98245dad0e3506"
server_key_version: "encryptionkey@0"
I20260811 11:50:32.826325 11735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "b82d4af0068c4b4d8bc9c5a6583398a2"
format_stamp: "Formatted at 2026-08-11 11:50:32 on dist-test-slave-bndk"
server_key: "22b60e12ba9c1162f51e87232a5f6b1c"
server_key_iv: "2fbd107abfcaf382cd98245dad0e3506"
server_key_version: "encryptionkey@0"
I20260811 11:50:32.830276 11735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 11:50:32.833021 11758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:32.834451 11735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 11:50:32.834580 11735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "b82d4af0068c4b4d8bc9c5a6583398a2"
format_stamp: "Formatted at 2026-08-11 11:50:32 on dist-test-slave-bndk"
server_key: "22b60e12ba9c1162f51e87232a5f6b1c"
server_key_iv: "2fbd107abfcaf382cd98245dad0e3506"
server_key_version: "encryptionkey@0"
I20260811 11:50:32.834694 11735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:32.839540 11735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:32.840212 11735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:32.840391 11735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:32.840986 11735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:32.841996 11735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:32.842046 11735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:32.842103 11735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:32.842131 11735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:32.850988 11735 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:35705
I20260811 11:50:32.851030 11942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:35705 every 8 connection(s)
I20260811 11:50:32.852012 11735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:50:32.854750 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 11735
I20260811 11:50:32.854883 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:50:32.855213 3957 external_mini_cluster.cc:1428] Setting key 089c243890b63b48df34ad0900754136
I20260811 11:50:32.857141 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:43673
--builtin_ntp_servers=127.3.221.84:46181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260811 11:50:32.858500 11701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:32.853883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:53091 (local address 127.3.221.126:43673)
0811 11:50:32.854019 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:32.854023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:32.855204 (+ 1181us) server_negotiation.cc:407] Connection header received
0811 11:50:32.856726 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:32.856730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:32.856773 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:32.856851 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:32.858043 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:32.858048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:32.858050 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:50:32.858064 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:32.858085 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:50:32.858093 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:32.858171 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:32.858315 (+ 144us) server_negotiation.cc:299] Negotiation successful
0811 11:50:32.858363 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 11:50:32.859444 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:32.854293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:43673 (local address 127.3.221.65:53091)
0811 11:50:32.855049 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:32.855094 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:32.856510 (+ 1416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:32.857033 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:32.857045 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:32.857140 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:32.857775 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:50:32.857776 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:32.857820 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:50:32.857897 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:32.858162 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:32.858171 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:32.858178 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 11:50:32.858352 (+ 174us) client_negotiation.cc:240] Negotiation successful
0811 11:50:32.858618 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":133,"threads_started":1}
I20260811 11:50:32.860785 11943 heartbeater.cc:347] Connected to a master server at 127.3.221.126:43673
I20260811 11:50:32.861050 11943 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:32.861595 11943 heartbeater.cc:511] Master 127.3.221.126:43673 requested a full tablet report, sending...
I20260811 11:50:32.863646 11624 ts_manager.cc:194] Registered new tserver with Master: b82d4af0068c4b4d8bc9c5a6583398a2 (127.3.221.65:35705)
W20260811 11:50:32.975986 11949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:32.976270 11949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:32.976325 11949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:32.980358 11949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:32.980424 11949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:32.980500 11949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:50:32.985381 11949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:46181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:43673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:32.986515 11949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:32.987919 11949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:32.997664 11956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:32.997675 11958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:32.997686 11962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:32.998024 11949 server_base.cc:1034] running on GCE node
I20260811 11:50:32.998544 11949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:32.999245 11949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:33.000412 11949 hybrid_clock.cc:648] HybridClock initialized: now 1786449033000386 us; error 41 us; skew 500 ppm
I20260811 11:50:33.002365 11949 webserver.cc:460] Webserver started at http://127.3.221.66:37471/ using document root <none> and password file <none>
I20260811 11:50:33.002976 11949 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:33.003044 11949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:33.003260 11949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:33.005576 11949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "c516b7bee1c048fc82ead03d65041ed5"
format_stamp: "Formatted at 2026-08-11 11:50:32 on dist-test-slave-bndk"
server_key: "c4b0437a823031502270376c46d30258"
server_key_iv: "9c75b4b5e0da54d48c78ce2e498478b4"
server_key_version: "encryptionkey@0"
I20260811 11:50:33.006187 11949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "c516b7bee1c048fc82ead03d65041ed5"
format_stamp: "Formatted at 2026-08-11 11:50:32 on dist-test-slave-bndk"
server_key: "c4b0437a823031502270376c46d30258"
server_key_iv: "9c75b4b5e0da54d48c78ce2e498478b4"
server_key_version: "encryptionkey@0"
I20260811 11:50:33.010040 11949 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 11:50:33.012470 11969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:33.013854 11949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 11:50:33.013974 11949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "c516b7bee1c048fc82ead03d65041ed5"
format_stamp: "Formatted at 2026-08-11 11:50:32 on dist-test-slave-bndk"
server_key: "c4b0437a823031502270376c46d30258"
server_key_iv: "9c75b4b5e0da54d48c78ce2e498478b4"
server_key_version: "encryptionkey@0"
I20260811 11:50:33.014081 11949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:33.018738 11949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:33.019373 11949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:33.019541 11949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:33.020116 11949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:33.021067 11949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:33.021116 11949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:33.021169 11949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:33.021195 11949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:33.030002 11949 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:37975
I20260811 11:50:33.030076 12164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:37975 every 8 connection(s)
I20260811 11:50:33.031037 11949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:50:33.035777 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 11949
I20260811 11:50:33.035928 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:50:33.036226 3957 external_mini_cluster.cc:1428] Setting key ee9a6950a81a1b7a085a1d466cf92872
I20260811 11:50:33.037509 11701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:33.032969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:33165 (local address 127.3.221.126:43673)
0811 11:50:33.033150 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:33.033154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:33.034244 (+ 1090us) server_negotiation.cc:407] Connection header received
0811 11:50:33.035640 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:33.035644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:33.035695 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:33.035789 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:33.037025 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:33.037029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:33.037031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:50:33.037045 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:33.037071 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:50:33.037081 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:33.037163 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:33.037333 (+ 170us) server_negotiation.cc:299] Negotiation successful
0811 11:50:33.037378 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 11:50:33.038455 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:43673
--builtin_ntp_servers=127.3.221.84:46181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260811 11:50:33.038487 12168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:33.033435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:43673 (local address 127.3.221.66:33165)
0811 11:50:33.034108 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:33.034152 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:33.035400 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:33.035980 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:33.035999 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:33.036105 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:33.036706 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:50:33.036708 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:33.036767 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:50:33.036861 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:33.037155 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:33.037161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:33.037164 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:50:33.037372 (+ 208us) client_negotiation.cc:240] Negotiation successful
0811 11:50:33.037604 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":114,"threads_started":1}
I20260811 11:50:33.039603 12165 heartbeater.cc:347] Connected to a master server at 127.3.221.126:43673
I20260811 11:50:33.039870 12165 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:33.040355 12165 heartbeater.cc:511] Master 127.3.221.126:43673 requested a full tablet report, sending...
I20260811 11:50:33.041448 11624 ts_manager.cc:194] Registered new tserver with Master: c516b7bee1c048fc82ead03d65041ed5 (127.3.221.66:37975)
W20260811 11:50:33.160995 12171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:33.161429 12171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:33.161509 12171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:33.167373 12171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:33.167448 12171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:33.167539 12171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:50:33.173427 12171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:46181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:43673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:33.174589 12171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:33.175987 12171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:33.185986 12176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:33.186072 12171 server_base.cc:1034] running on GCE node
W20260811 11:50:33.185986 12178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:33.186015 12180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:33.186616 12171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:33.187266 12171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:33.188438 12171 hybrid_clock.cc:648] HybridClock initialized: now 1786449033188417 us; error 34 us; skew 500 ppm
I20260811 11:50:33.190297 12171 webserver.cc:460] Webserver started at http://127.3.221.67:44835/ using document root <none> and password file <none>
I20260811 11:50:33.190901 12171 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:33.190973 12171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:33.191172 12171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:33.193380 12171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "2c789df82962446ab5118bc13fbcbb11"
format_stamp: "Formatted at 2026-08-11 11:50:33 on dist-test-slave-bndk"
server_key: "16a8c7eafb3c88ee2cc8cda8348daf6b"
server_key_iv: "d2cc1aad3be736325462b46bfa022dde"
server_key_version: "encryptionkey@0"
I20260811 11:50:33.193956 12171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "2c789df82962446ab5118bc13fbcbb11"
format_stamp: "Formatted at 2026-08-11 11:50:33 on dist-test-slave-bndk"
server_key: "16a8c7eafb3c88ee2cc8cda8348daf6b"
server_key_iv: "d2cc1aad3be736325462b46bfa022dde"
server_key_version: "encryptionkey@0"
I20260811 11:50:33.198072 12171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 11:50:33.200652 12186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:33.202068 12171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 11:50:33.202186 12171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "2c789df82962446ab5118bc13fbcbb11"
format_stamp: "Formatted at 2026-08-11 11:50:33 on dist-test-slave-bndk"
server_key: "16a8c7eafb3c88ee2cc8cda8348daf6b"
server_key_iv: "d2cc1aad3be736325462b46bfa022dde"
server_key_version: "encryptionkey@0"
I20260811 11:50:33.202284 12171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:33.207021 12171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:33.207631 12171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:33.207774 12171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:33.208312 12171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:33.209281 12171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:33.209336 12171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:33.209379 12171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:33.209399 12171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:33.218278 12171 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:35367
I20260811 11:50:33.218307 12372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:35367 every 8 connection(s)
I20260811 11:50:33.219229 12171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:50:33.221516 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 12171
I20260811 11:50:33.221629 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:50:33.221897 3957 external_mini_cluster.cc:1428] Setting key 3c82edc0d116a2c406e2e7821ea78541
I20260811 11:50:33.225577 11701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:33.221040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:50929 (local address 127.3.221.126:43673)
0811 11:50:33.221251 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:33.221257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:33.222457 (+ 1200us) server_negotiation.cc:407] Connection header received
0811 11:50:33.223975 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:33.223978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:33.224023 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:33.224094 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:33.225112 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:33.225117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:33.225119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:50:33.225132 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:33.225154 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:50:33.225162 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:33.225261 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:33.225398 (+ 137us) server_negotiation.cc:299] Negotiation successful
0811 11:50:33.225445 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260811 11:50:33.226472 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:33.221523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:43673 (local address 127.3.221.67:50929)
0811 11:50:33.222301 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:33.222356 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:33.223745 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:33.224265 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:33.224273 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:33.224365 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:33.224860 (+ 495us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:50:33.224861 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:33.224901 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:50:33.224972 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:33.225260 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:33.225269 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:33.225275 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 11:50:33.225442 (+ 167us) client_negotiation.cc:240] Negotiation successful
0811 11:50:33.225694 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":147,"threads_started":1}
I20260811 11:50:33.227516 12373 heartbeater.cc:347] Connected to a master server at 127.3.221.126:43673
I20260811 11:50:33.227747 12373 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:33.228181 12373 heartbeater.cc:511] Master 127.3.221.126:43673 requested a full tablet report, sending...
I20260811 11:50:33.229169 11624 ts_manager.cc:194] Registered new tserver with Master: 2c789df82962446ab5118bc13fbcbb11 (127.3.221.67:35367)
I20260811 11:50:33.235690 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:50:33.239441 11701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:33.237664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44070 (local address 127.3.221.126:43673)
0811 11:50:33.237809 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:33.237814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:33.237966 (+ 152us) server_negotiation.cc:407] Connection header received
0811 11:50:33.238118 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:33.238120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:33.238156 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:33.238225 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:33.239078 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:33.239082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:33.239085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:50:33.239099 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:33.239124 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:50:33.239133 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:33.239209 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:33.239270 (+ 61us) server_negotiation.cc:299] Negotiation successful
0811 11:50:33.239304 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 11:50:33.242187 11624 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:50:33.245303 11624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:50:33.257206 12387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:33.253702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:35705 (local address 127.0.0.1:37268)
0811 11:50:33.254313 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:33.254367 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:33.254483 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:33.255635 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:33.255638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:33.255652 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:33.256168 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:50:33.256170 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:33.256193 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:50:33.256274 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:33.256999 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:33.257002 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:33.257005 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:50:33.257051 (+ 46us) client_negotiation.cc:240] Negotiation successful
0811 11:50:33.257076 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":91,"threads_started":1}
I20260811 11:50:33.257210 12392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:33.254462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:37975 (local address 127.0.0.1:59926)
0811 11:50:33.254884 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:33.254896 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:33.254965 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:33.255765 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:33.255768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:33.255785 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:33.256287 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:50:33.256289 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:33.256301 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:50:33.256332 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:33.256810 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:33.256821 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:33.256827 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 11:50:33.256957 (+ 130us) client_negotiation.cc:240] Negotiation successful
0811 11:50:33.257054 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":7,"thread_start_us":89,"threads_started":1}
I20260811 11:50:33.257373 12394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:33.254753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59926 (local address 127.3.221.66:37975)
0811 11:50:33.255207 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:33.255223 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:33.255246 (+ 23us) server_negotiation.cc:407] Connection header received
0811 11:50:33.255292 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:33.255297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:33.255589 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:33.256021 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:33.256642 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:33.256648 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:33.256659 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:50:33.256669 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:33.256707 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:50:33.256725 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:33.256832 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:33.257092 (+ 260us) server_negotiation.cc:299] Negotiation successful
0811 11:50:33.257177 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":146,"threads_started":1}
I20260811 11:50:33.257867 12391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:33.254220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36194 (local address 127.3.221.67:35367)
0811 11:50:33.254740 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:33.254754 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:33.254961 (+ 207us) server_negotiation.cc:407] Connection header received
0811 11:50:33.255044 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:33.255051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:33.255267 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:33.255479 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:33.256591 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:33.256598 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:33.256612 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:50:33.256625 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:33.256680 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:50:33.256716 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:33.256831 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:33.257638 (+ 807us) server_negotiation.cc:299] Negotiation successful
0811 11:50:33.257715 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":154,"threads_started":1}
I20260811 11:50:33.258061 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:33.253819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:35367 (local address 127.0.0.1:36194)
0811 11:50:33.254749 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:33.254762 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:33.254846 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:33.255432 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:33.255438 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:33.255517 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:33.256414 (+ 897us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:50:33.256416 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:33.256426 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:50:33.256451 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:33.256903 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:33.256906 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:33.256909 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:50:33.256982 (+ 73us) client_negotiation.cc:240] Negotiation successful
0811 11:50:33.257037 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"thread_start_us":119,"threads_started":1}
I20260811 11:50:33.258461 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:33.253839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37268 (local address 127.3.221.65:35705)
0811 11:50:33.255208 (+ 1369us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:33.255217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:33.255238 (+ 21us) server_negotiation.cc:407] Connection header received
0811 11:50:33.255281 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:33.255285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:33.255487 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:33.255838 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:33.256805 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:33.256810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:33.256818 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:50:33.256828 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:33.256875 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:50:33.256901 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:33.258149 (+ 1248us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:33.258197 (+ 48us) server_negotiation.cc:299] Negotiation successful
0811 11:50:33.258268 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1273,"thread_start_us":118,"threads_started":1}
I20260811 11:50:33.260465 12050 tablet_service.cc:1467] Processing CreateTablet for tablet 191af0766f3f44ed98cd0fcfa329d2d3 (DEFAULT_TABLE table=test-table [id=6f1aaec5849a44219210eb13094cf317]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:33.260947 12263 tablet_service.cc:1467] Processing CreateTablet for tablet 191af0766f3f44ed98cd0fcfa329d2d3 (DEFAULT_TABLE table=test-table [id=6f1aaec5849a44219210eb13094cf317]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:33.260947 11827 tablet_service.cc:1467] Processing CreateTablet for tablet 191af0766f3f44ed98cd0fcfa329d2d3 (DEFAULT_TABLE table=test-table [id=6f1aaec5849a44219210eb13094cf317]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:33.261682 12050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 191af0766f3f44ed98cd0fcfa329d2d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:33.262017 12263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 191af0766f3f44ed98cd0fcfa329d2d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:33.262017 11827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 191af0766f3f44ed98cd0fcfa329d2d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:33.268033 12395 tablet_bootstrap.cc:492] T 191af0766f3f44ed98cd0fcfa329d2d3 P c516b7bee1c048fc82ead03d65041ed5: Bootstrap starting.
I20260811 11:50:33.269543 12397 tablet_bootstrap.cc:492] T 191af0766f3f44ed98cd0fcfa329d2d3 P 2c789df82962446ab5118bc13fbcbb11: Bootstrap starting.
I20260811 11:50:33.270630 12396 tablet_bootstrap.cc:492] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2: Bootstrap starting.
I20260811 11:50:33.270682 12395 tablet_bootstrap.cc:654] T 191af0766f3f44ed98cd0fcfa329d2d3 P c516b7bee1c048fc82ead03d65041ed5: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:33.271746 12395 log.cc:824] T 191af0766f3f44ed98cd0fcfa329d2d3 P c516b7bee1c048fc82ead03d65041ed5: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:33.271828 12397 tablet_bootstrap.cc:654] T 191af0766f3f44ed98cd0fcfa329d2d3 P 2c789df82962446ab5118bc13fbcbb11: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:33.272512 12396 tablet_bootstrap.cc:654] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:33.272579 12397 log.cc:824] T 191af0766f3f44ed98cd0fcfa329d2d3 P 2c789df82962446ab5118bc13fbcbb11: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:33.273304 12396 log.cc:824] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:33.274118 12395 tablet_bootstrap.cc:492] T 191af0766f3f44ed98cd0fcfa329d2d3 P c516b7bee1c048fc82ead03d65041ed5: No bootstrap required, opened a new log
I20260811 11:50:33.274336 12395 ts_tablet_manager.cc:1397] T 191af0766f3f44ed98cd0fcfa329d2d3 P c516b7bee1c048fc82ead03d65041ed5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 11:50:33.274713 12397 tablet_bootstrap.cc:492] T 191af0766f3f44ed98cd0fcfa329d2d3 P 2c789df82962446ab5118bc13fbcbb11: No bootstrap required, opened a new log
I20260811 11:50:33.274907 12397 ts_tablet_manager.cc:1397] T 191af0766f3f44ed98cd0fcfa329d2d3 P 2c789df82962446ab5118bc13fbcbb11: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 11:50:33.274968 12396 tablet_bootstrap.cc:492] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2: No bootstrap required, opened a new log
I20260811 11:50:33.275136 12396 ts_tablet_manager.cc:1397] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260811 11:50:33.278877 12396 raft_consensus.cc:348] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b82d4af0068c4b4d8bc9c5a6583398a2" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35705 } } peers { permanent_uuid: "2c789df82962446ab5118bc13fbcbb11" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 35367 } } peers { permanent_uuid: "c516b7bee1c048fc82ead03d65041ed5" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 37975 } }
I20260811 11:50:33.279145 12396 raft_consensus.cc:374] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:33.279198 12396 raft_consensus.cc:729] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b82d4af0068c4b4d8bc9c5a6583398a2, State: Initialized, Role: FOLLOWER
I20260811 11:50:33.279086 12395 raft_consensus.cc:348] T 191af0766f3f44ed98cd0fcfa329d2d3 P c516b7bee1c048fc82ead03d65041ed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b82d4af0068c4b4d8bc9c5a6583398a2" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35705 } } peers { permanent_uuid: "2c789df82962446ab5118bc13fbcbb11" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 35367 } } peers { permanent_uuid: "c516b7bee1c048fc82ead03d65041ed5" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 37975 } }
I20260811 11:50:33.279400 12395 raft_consensus.cc:374] T 191af0766f3f44ed98cd0fcfa329d2d3 P c516b7bee1c048fc82ead03d65041ed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:33.279241 12397 raft_consensus.cc:348] T 191af0766f3f44ed98cd0fcfa329d2d3 P 2c789df82962446ab5118bc13fbcbb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b82d4af0068c4b4d8bc9c5a6583398a2" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35705 } } peers { permanent_uuid: "2c789df82962446ab5118bc13fbcbb11" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 35367 } } peers { permanent_uuid: "c516b7bee1c048fc82ead03d65041ed5" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 37975 } }
I20260811 11:50:33.279461 12395 raft_consensus.cc:729] T 191af0766f3f44ed98cd0fcfa329d2d3 P c516b7bee1c048fc82ead03d65041ed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c516b7bee1c048fc82ead03d65041ed5, State: Initialized, Role: FOLLOWER
I20260811 11:50:33.279479 12397 raft_consensus.cc:374] T 191af0766f3f44ed98cd0fcfa329d2d3 P 2c789df82962446ab5118bc13fbcbb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:33.279523 12397 raft_consensus.cc:729] T 191af0766f3f44ed98cd0fcfa329d2d3 P 2c789df82962446ab5118bc13fbcbb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c789df82962446ab5118bc13fbcbb11, State: Initialized, Role: FOLLOWER
I20260811 11:50:33.279727 12396 consensus_queue.cc:260] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b82d4af0068c4b4d8bc9c5a6583398a2" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35705 } } peers { permanent_uuid: "2c789df82962446ab5118bc13fbcbb11" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 35367 } } peers { permanent_uuid: "c516b7bee1c048fc82ead03d65041ed5" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 37975 } }
I20260811 11:50:33.279938 12395 consensus_queue.cc:260] T 191af0766f3f44ed98cd0fcfa329d2d3 P c516b7bee1c048fc82ead03d65041ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b82d4af0068c4b4d8bc9c5a6583398a2" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35705 } } peers { permanent_uuid: "2c789df82962446ab5118bc13fbcbb11" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 35367 } } peers { permanent_uuid: "c516b7bee1c048fc82ead03d65041ed5" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 37975 } }
I20260811 11:50:33.279942 12397 consensus_queue.cc:260] T 191af0766f3f44ed98cd0fcfa329d2d3 P 2c789df82962446ab5118bc13fbcbb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b82d4af0068c4b4d8bc9c5a6583398a2" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35705 } } peers { permanent_uuid: "2c789df82962446ab5118bc13fbcbb11" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 35367 } } peers { permanent_uuid: "c516b7bee1c048fc82ead03d65041ed5" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 37975 } }
I20260811 11:50:33.280608 11943 heartbeater.cc:503] Master 127.3.221.126:43673 was elected leader, sending a full tablet report...
I20260811 11:50:33.280869 12396 ts_tablet_manager.cc:1428] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260811 11:50:33.280941 12397 ts_tablet_manager.cc:1428] T 191af0766f3f44ed98cd0fcfa329d2d3 P 2c789df82962446ab5118bc13fbcbb11: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260811 11:50:33.280941 12395 ts_tablet_manager.cc:1428] T 191af0766f3f44ed98cd0fcfa329d2d3 P c516b7bee1c048fc82ead03d65041ed5: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260811 11:50:33.281333 12165 heartbeater.cc:503] Master 127.3.221.126:43673 was elected leader, sending a full tablet report...
I20260811 11:50:33.281605 12373 heartbeater.cc:503] Master 127.3.221.126:43673 was elected leader, sending a full tablet report...
W20260811 11:50:33.281982 12166 tablet.cc:2369] T 191af0766f3f44ed98cd0fcfa329d2d3 P c516b7bee1c048fc82ead03d65041ed5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:50:33.286880 12401 raft_consensus.cc:482] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:50:33.287053 12401 raft_consensus.cc:504] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b82d4af0068c4b4d8bc9c5a6583398a2" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35705 } } peers { permanent_uuid: "2c789df82962446ab5118bc13fbcbb11" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 35367 } } peers { permanent_uuid: "c516b7bee1c048fc82ead03d65041ed5" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 37975 } }
I20260811 11:50:33.288146 12401 leader_election.cc:288] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c789df82962446ab5118bc13fbcbb11 (127.3.221.67:35367), c516b7bee1c048fc82ead03d65041ed5 (127.3.221.66:37975)
I20260811 11:50:33.290009 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:33.288406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:37975 (local address 127.3.221.65:38783)
0811 11:50:33.288524 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:33.288541 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:33.288612 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:33.288878 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:33.288881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:33.288897 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:33.289439 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:50:33.289441 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:33.289454 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:50:33.289482 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:33.289746 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:33.289749 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:33.289752 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:50:33.289842 (+ 90us) client_negotiation.cc:240] Negotiation successful
0811 11:50:33.289881 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260811 11:50:33.290009 12394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:33.288482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:38783 (local address 127.3.221.66:37975)
0811 11:50:33.288601 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:33.288605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:33.288618 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:50:33.288710 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:33.288713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:33.288752 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:33.288836 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:33.289608 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:33.289612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:33.289614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:50:33.289629 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:33.289651 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:50:33.289661 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:33.289753 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:33.289842 (+ 89us) server_negotiation.cc:299] Negotiation successful
0811 11:50:33.289881 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 11:50:33.290463 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:33.288406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:35367 (local address 127.3.221.65:59469)
0811 11:50:33.289016 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:33.289028 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:33.289130 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:33.289434 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:33.289438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:33.289455 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:33.289873 (+ 418us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:50:33.289875 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:33.289886 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:50:33.289912 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:33.290201 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:33.290204 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:33.290207 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 11:50:33.290279 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 11:50:33.290311 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":263,"thread_start_us":117,"threads_started":1}
I20260811 11:50:33.290483 12391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:33.288485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:59469 (local address 127.3.221.67:35367)
0811 11:50:33.288708 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:33.288712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:33.289120 (+ 408us) server_negotiation.cc:407] Connection header received
0811 11:50:33.289211 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:33.289249 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:33.289318 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:33.289395 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:33.290026 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:33.290030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:33.290032 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:50:33.290045 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:33.290064 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:50:33.290075 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:33.290159 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:33.290327 (+ 168us) server_negotiation.cc:299] Negotiation successful
0811 11:50:33.290369 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260811 11:50:33.290771 12086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191af0766f3f44ed98cd0fcfa329d2d3" candidate_uuid: "b82d4af0068c4b4d8bc9c5a6583398a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c516b7bee1c048fc82ead03d65041ed5" is_pre_election: true
I20260811 11:50:33.291040 12086 raft_consensus.cc:2454] T 191af0766f3f44ed98cd0fcfa329d2d3 P c516b7bee1c048fc82ead03d65041ed5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b82d4af0068c4b4d8bc9c5a6583398a2 in term 0.
I20260811 11:50:33.291531 11768 leader_election.cc:302] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b82d4af0068c4b4d8bc9c5a6583398a2, c516b7bee1c048fc82ead03d65041ed5; no voters:
I20260811 11:50:33.291836 12300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191af0766f3f44ed98cd0fcfa329d2d3" candidate_uuid: "b82d4af0068c4b4d8bc9c5a6583398a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c789df82962446ab5118bc13fbcbb11" is_pre_election: true
I20260811 11:50:33.292111 12300 raft_consensus.cc:2454] T 191af0766f3f44ed98cd0fcfa329d2d3 P 2c789df82962446ab5118bc13fbcbb11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b82d4af0068c4b4d8bc9c5a6583398a2 in term 0.
I20260811 11:50:33.292390 12401 raft_consensus.cc:2781] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:50:33.292496 12401 raft_consensus.cc:482] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:50:33.292552 12401 raft_consensus.cc:3037] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:33.293712 12401 raft_consensus.cc:504] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b82d4af0068c4b4d8bc9c5a6583398a2" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35705 } } peers { permanent_uuid: "2c789df82962446ab5118bc13fbcbb11" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 35367 } } peers { permanent_uuid: "c516b7bee1c048fc82ead03d65041ed5" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 37975 } }
I20260811 11:50:33.294061 12401 leader_election.cc:288] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2 [CANDIDATE]: Term 1 election: Requested vote from peers 2c789df82962446ab5118bc13fbcbb11 (127.3.221.67:35367), c516b7bee1c048fc82ead03d65041ed5 (127.3.221.66:37975)
I20260811 11:50:33.294438 12300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191af0766f3f44ed98cd0fcfa329d2d3" candidate_uuid: "b82d4af0068c4b4d8bc9c5a6583398a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c789df82962446ab5118bc13fbcbb11"
I20260811 11:50:33.294440 12086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191af0766f3f44ed98cd0fcfa329d2d3" candidate_uuid: "b82d4af0068c4b4d8bc9c5a6583398a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c516b7bee1c048fc82ead03d65041ed5"
I20260811 11:50:33.294584 12300 raft_consensus.cc:3037] T 191af0766f3f44ed98cd0fcfa329d2d3 P 2c789df82962446ab5118bc13fbcbb11 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:33.294584 12086 raft_consensus.cc:3037] T 191af0766f3f44ed98cd0fcfa329d2d3 P c516b7bee1c048fc82ead03d65041ed5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:33.295956 12300 raft_consensus.cc:2454] T 191af0766f3f44ed98cd0fcfa329d2d3 P 2c789df82962446ab5118bc13fbcbb11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b82d4af0068c4b4d8bc9c5a6583398a2 in term 1.
I20260811 11:50:33.295956 12086 raft_consensus.cc:2454] T 191af0766f3f44ed98cd0fcfa329d2d3 P c516b7bee1c048fc82ead03d65041ed5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b82d4af0068c4b4d8bc9c5a6583398a2 in term 1.
I20260811 11:50:33.296314 11762 leader_election.cc:302] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2c789df82962446ab5118bc13fbcbb11, b82d4af0068c4b4d8bc9c5a6583398a2, c516b7bee1c048fc82ead03d65041ed5; no voters:
I20260811 11:50:33.296526 12401 raft_consensus.cc:2781] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:33.296787 12401 raft_consensus.cc:686] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2 [term 1 LEADER]: Becoming Leader. State: Replica: b82d4af0068c4b4d8bc9c5a6583398a2, State: Running, Role: LEADER
I20260811 11:50:33.297137 12401 consensus_queue.cc:237] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b82d4af0068c4b4d8bc9c5a6583398a2" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35705 } } peers { permanent_uuid: "2c789df82962446ab5118bc13fbcbb11" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 35367 } } peers { permanent_uuid: "c516b7bee1c048fc82ead03d65041ed5" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 37975 } }
I20260811 11:50:33.300715 11623 catalog_manager.cc:5495] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to b82d4af0068c4b4d8bc9c5a6583398a2 (127.3.221.65). New cstate: current_term: 1 leader_uuid: "b82d4af0068c4b4d8bc9c5a6583398a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b82d4af0068c4b4d8bc9c5a6583398a2" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c789df82962446ab5118bc13fbcbb11" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 35367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c516b7bee1c048fc82ead03d65041ed5" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260811 11:50:33.322758 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:33.321266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37272 (local address 127.3.221.65:35705)
0811 11:50:33.321398 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:33.321403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:33.321418 (+ 15us) server_negotiation.cc:407] Connection header received
0811 11:50:33.321591 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:33.321593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:33.321655 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:33.321729 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:33.322361 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:33.322364 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:33.322367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:50:33.322380 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:33.322399 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:50:33.322407 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:33.322487 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:33.322561 (+ 74us) server_negotiation.cc:299] Negotiation successful
0811 11:50:33.322605 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 11:50:33.328893 12086 raft_consensus.cc:1261] T 191af0766f3f44ed98cd0fcfa329d2d3 P c516b7bee1c048fc82ead03d65041ed5 [term 1 FOLLOWER]: Refusing update from remote peer b82d4af0068c4b4d8bc9c5a6583398a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:33.328893 12300 raft_consensus.cc:1261] T 191af0766f3f44ed98cd0fcfa329d2d3 P 2c789df82962446ab5118bc13fbcbb11 [term 1 FOLLOWER]: Refusing update from remote peer b82d4af0068c4b4d8bc9c5a6583398a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:33.329677 12412 consensus_queue.cc:1035] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c789df82962446ab5118bc13fbcbb11" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 35367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:33.329872 12401 consensus_queue.cc:1035] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c516b7bee1c048fc82ead03d65041ed5" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:33.337301 12417 mvcc.cc:205] Tried to move back new op lower bound from 7317295240507527168 to 7317295240388194304. Current Snapshot: MvccSnapshot[applied={T|T < 7317295240507527168}]
I20260811 11:50:33.339361 12418 mvcc.cc:205] Tried to move back new op lower bound from 7317295240507527168 to 7317295240388194304. Current Snapshot: MvccSnapshot[applied={T|T < 7317295240507527168}]
I20260811 11:50:33.347481 11622 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:33.347662 11622 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:33.350262 11622 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10677dc2e4a24767bcd3fac9fbced12d: Sending DeleteTablet for 3 replicas of tablet 191af0766f3f44ed98cd0fcfa329d2d3
I20260811 11:50:33.351181 12263 tablet_service.cc:1514] Processing DeleteTablet for tablet 191af0766f3f44ed98cd0fcfa329d2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:33 UTC) from {username='slave'} at 127.0.0.1:36194
I20260811 11:50:33.351255 12050 tablet_service.cc:1514] Processing DeleteTablet for tablet 191af0766f3f44ed98cd0fcfa329d2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:33 UTC) from {username='slave'} at 127.0.0.1:59926
I20260811 11:50:33.351356 11827 tablet_service.cc:1514] Processing DeleteTablet for tablet 191af0766f3f44ed98cd0fcfa329d2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:33 UTC) from {username='slave'} at 127.0.0.1:37268
I20260811 11:50:33.351804 12435 tablet_replica.cc:330] T 191af0766f3f44ed98cd0fcfa329d2d3 P b82d4af0068c4b4d8bc9c5a6583398a2: stopping tablet replica
I20260811 11:50:33.351944 12433 tablet_replica.cc:330] T 191af0766f3f44ed98cd0fcfa329d2d3 P c516b7bee1c048fc82ead03d65041ed5: stopping tablet replica
I20260811 11:50:33.352166 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 11735
I20260811 11:50:33.352283 12433 raft_consensus.cc:2229] T 191af0766f3f44ed98cd0fcfa329d2d3 P c516b7bee1c048fc82ead03d65041ed5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:33.354815 12433 raft_consensus.cc:2258] T 191af0766f3f44ed98cd0fcfa329d2d3 P c516b7bee1c048fc82ead03d65041ed5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:33.355655 12436 tablet_replica.cc:330] T 191af0766f3f44ed98cd0fcfa329d2d3 P 2c789df82962446ab5118bc13fbcbb11: stopping tablet replica
I20260811 11:50:33.355973 12436 raft_consensus.cc:2229] T 191af0766f3f44ed98cd0fcfa329d2d3 P 2c789df82962446ab5118bc13fbcbb11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:33.356254 12436 raft_consensus.cc:2258] T 191af0766f3f44ed98cd0fcfa329d2d3 P 2c789df82962446ab5118bc13fbcbb11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:33.359135 12433 ts_tablet_manager.cc:1905] T 191af0766f3f44ed98cd0fcfa329d2d3 P c516b7bee1c048fc82ead03d65041ed5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:33.361272 12436 ts_tablet_manager.cc:1905] T 191af0766f3f44ed98cd0fcfa329d2d3 P 2c789df82962446ab5118bc13fbcbb11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:33.362001 12433 ts_tablet_manager.cc:1918] T 191af0766f3f44ed98cd0fcfa329d2d3 P c516b7bee1c048fc82ead03d65041ed5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:50:33.362110 12433 log.cc:1196] T 191af0766f3f44ed98cd0fcfa329d2d3 P c516b7bee1c048fc82ead03d65041ed5: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/191af0766f3f44ed98cd0fcfa329d2d3
I20260811 11:50:33.362444 12433 ts_tablet_manager.cc:1939] T 191af0766f3f44ed98cd0fcfa329d2d3 P c516b7bee1c048fc82ead03d65041ed5: Deleting consensus metadata
I20260811 11:50:33.363976 11605 catalog_manager.cc:4841] TS c516b7bee1c048fc82ead03d65041ed5 (127.3.221.66:37975): tablet 191af0766f3f44ed98cd0fcfa329d2d3 (table test-table [id=6f1aaec5849a44219210eb13094cf317]) successfully deleted
I20260811 11:50:33.364048 12436 ts_tablet_manager.cc:1918] T 191af0766f3f44ed98cd0fcfa329d2d3 P 2c789df82962446ab5118bc13fbcbb11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:50:33.364138 12436 log.cc:1196] T 191af0766f3f44ed98cd0fcfa329d2d3 P 2c789df82962446ab5118bc13fbcbb11: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/191af0766f3f44ed98cd0fcfa329d2d3
I20260811 11:50:33.364416 12436 ts_tablet_manager.cc:1939] T 191af0766f3f44ed98cd0fcfa329d2d3 P 2c789df82962446ab5118bc13fbcbb11: Deleting consensus metadata
I20260811 11:50:33.365557 11602 catalog_manager.cc:4841] TS 2c789df82962446ab5118bc13fbcbb11 (127.3.221.67:35367): tablet 191af0766f3f44ed98cd0fcfa329d2d3 (table test-table [id=6f1aaec5849a44219210eb13094cf317]) successfully deleted
I20260811 11:50:33.366490 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 11949
W20260811 11:50:33.366473 11605 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.221.65:35705 (error 108)
I20260811 11:50:33.372431 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:33.372039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:35705 (local address 127.0.0.1:37288)
0811 11:50:33.372199 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:33.372317 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.65:35705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260811 11:50:33.372926 11605 catalog_manager.cc:4584] TS b82d4af0068c4b4d8bc9c5a6583398a2 (127.3.221.65:35705): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 191af0766f3f44ed98cd0fcfa329d2d3: Network error: Client connection negotiation failed: client connection to 127.3.221.65:35705: connect: Connection refused (error 111)
I20260811 11:50:33.379279 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 12171
I20260811 11:50:33.395576 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 11582
2026-08-11T11:50:33Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (997 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260811 11:50:33.427680 3957 mini_oidc.cc:152] Starting JWKS server
I20260811 11:50:33.429633 3957 webserver.cc:460] Webserver started at https://127.0.0.1:43521/ using document root <none> and password file <none>
I20260811 11:50:33.429812 3957 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 11:50:33.430083 3957 webserver.cc:460] Webserver started at http://127.0.0.1:38343/ using document root <none> and password file <none>
2026-08-11T11:50:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:50:33Z Disabled control of system clock
I20260811 11:50:33.440665 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:38177
--webserver_interface=127.3.221.126
--webserver_port=0
--builtin_ntp_servers=127.3.221.84:44935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:38177
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43521/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786449004114841-3957-0/testchainca.cert with env {}
W20260811 11:50:33.555475 12445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:33.555773 12445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:33.555814 12445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:33.559952 12445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:50:33.560025 12445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:33.560040 12445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:33.560056 12445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:50:33.560070 12445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 11:50:33.560086 12445 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 11:50:33.560097 12445 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43521/jwks.json
I20260811 11:50:33.565660 12445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:44935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:38177
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:38177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43521/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786449004114841-3957-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:33.566848 12445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:33.567860 12445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:33.576715 12457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:33.576924 12445 server_base.cc:1034] running on GCE node
W20260811 11:50:33.576720 12458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:33.576728 12460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:33.577785 12445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:33.579012 12445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:33.580206 12445 hybrid_clock.cc:648] HybridClock initialized: now 1786449033580185 us; error 45 us; skew 500 ppm
I20260811 11:50:33.582265 12445 webserver.cc:460] Webserver started at http://127.3.221.126:35803/ using document root <none> and password file <none>
I20260811 11:50:33.582864 12445 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:33.582939 12445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:33.583161 12445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:33.585356 12445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "90e05afb8a624c75a104edcf8e6ee00b"
format_stamp: "Formatted at 2026-08-11 11:50:33 on dist-test-slave-bndk"
server_key: "1986ccdcf958c35485ed947c8876c33c"
server_key_iv: "082a038370e00473b2afee3a7d608c4f"
server_key_version: "encryptionkey@0"
I20260811 11:50:33.585980 12445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "90e05afb8a624c75a104edcf8e6ee00b"
format_stamp: "Formatted at 2026-08-11 11:50:33 on dist-test-slave-bndk"
server_key: "1986ccdcf958c35485ed947c8876c33c"
server_key_iv: "082a038370e00473b2afee3a7d608c4f"
server_key_version: "encryptionkey@0"
I20260811 11:50:33.590080 12445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 11:50:33.592870 12468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:33.594238 12445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 11:50:33.594378 12445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "90e05afb8a624c75a104edcf8e6ee00b"
format_stamp: "Formatted at 2026-08-11 11:50:33 on dist-test-slave-bndk"
server_key: "1986ccdcf958c35485ed947c8876c33c"
server_key_iv: "082a038370e00473b2afee3a7d608c4f"
server_key_version: "encryptionkey@0"
I20260811 11:50:33.594499 12445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:33.614135 12445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:33.614944 12445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:33.615149 12445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:33.622448 12445 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:38177
I20260811 11:50:33.622464 12566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:38177 every 8 connection(s)
I20260811 11:50:33.623467 12445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:50:33.626466 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:33.627298 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 12445
I20260811 11:50:33.627440 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:50:33.627763 3957 external_mini_cluster.cc:1428] Setting key 33ace6f6d372e97eafc7be56a25ce916
I20260811 11:50:33.633137 12567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b: Bootstrap starting.
I20260811 11:50:33.635928 12567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:33.636756 12567 log.cc:824] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:33.637694 12574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:33.629286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36128 (local address 127.3.221.126:38177)
0811 11:50:33.630010 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:33.630026 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:33.630065 (+ 39us) server_negotiation.cc:407] Connection header received
0811 11:50:33.630871 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:33.630896 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:33.631371 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:33.631670 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:33.632037 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:33.632964 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:33.634185 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:33.634699 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:33.634830 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:33.635692 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:33.635698 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:33.635711 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:50:33.636122 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:33.636187 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:50:33.636210 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:33.636294 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:33.636442 (+ 148us) server_negotiation.cc:299] Negotiation successful
0811 11:50:33.636714 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":140,"threads_started":1}
I20260811 11:50:33.639003 12567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b: No bootstrap required, opened a new log
I20260811 11:50:33.640614 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:50:33.643028 12567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e05afb8a624c75a104edcf8e6ee00b" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38177 } }
I20260811 11:50:33.643280 12567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:33.643338 12567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90e05afb8a624c75a104edcf8e6ee00b, State: Initialized, Role: FOLLOWER
I20260811 11:50:33.643807 12567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e05afb8a624c75a104edcf8e6ee00b" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38177 } }
I20260811 11:50:33.643981 12567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:50:33.644053 12567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:50:33.644152 12567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:33.645102 12567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e05afb8a624c75a104edcf8e6ee00b" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38177 } }
I20260811 11:50:33.645457 12567 leader_election.cc:302] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90e05afb8a624c75a104edcf8e6ee00b; no voters:
I20260811 11:50:33.645763 12567 leader_election.cc:288] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:50:33.645888 12576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:33.646131 12576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b [term 1 LEADER]: Becoming Leader. State: Replica: 90e05afb8a624c75a104edcf8e6ee00b, State: Running, Role: LEADER
I20260811 11:50:33.646446 12576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e05afb8a624c75a104edcf8e6ee00b" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38177 } }
I20260811 11:50:33.646993 12567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:50:33.648253 12579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90e05afb8a624c75a104edcf8e6ee00b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e05afb8a624c75a104edcf8e6ee00b" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38177 } } }
I20260811 11:50:33.648285 12580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90e05afb8a624c75a104edcf8e6ee00b. Latest consensus state: current_term: 1 leader_uuid: "90e05afb8a624c75a104edcf8e6ee00b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e05afb8a624c75a104edcf8e6ee00b" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38177 } } }
I20260811 11:50:33.648392 12579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:33.648391 12580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:33.648778 12596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:50:33.652206 12596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:50:33.657774 12596 catalog_manager.cc:1347] Generated new cluster ID: 176301b7787e434d89ed8cb69efa6753
I20260811 11:50:33.657859 12596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:50:33.664611 12596 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:50:33.665639 12596 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:50:33.682740 12596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90e05afb8a624c75a104edcf8e6ee00b: Generated new TSK 0
I20260811 11:50:33.683544 12596 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260811 11:50:33.723278 12623 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260811 11:50:33.723456 12623 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260811 11:50:33.724146 12574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:33.703838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36134 (local address 127.3.221.126:38177)
0811 11:50:33.703991 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:33.703996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:33.704263 (+ 267us) server_negotiation.cc:407] Connection header received
0811 11:50:33.704480 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:33.704483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:33.704547 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:33.704672 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0811 11:50:33.704889 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:33.705790 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:33.706645 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:33.707011 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:33.707111 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:33.707419 (+ 308us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 11:50:33.723662 (+ 16243us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0811 11:50:33.723980 (+ 318us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36134: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260811 11:50:33.724242 12574 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36134: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260811 11:50:33.725566 12620 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:33.703734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:38177 (local address 127.0.0.1:36134)
0811 11:50:33.704175 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:33.704188 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:33.704306 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:33.704674 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:33.704676 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:33.704682 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0811 11:50:33.704791 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:33.704796 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:33.705905 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:33.705908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:33.706538 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:33.706548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:33.707107 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:33.707110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:33.707172 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:33.707222 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 11:50:33.723910 (+ 16688us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0811 11:50:33.724180 (+ 270us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.221.126:38177: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":109,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.221.126:38177: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260811 11:50:33.726251 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 12445
2026-08-11T11:50:33Z chronyd exiting
I20260811 11:50:33.772903 3957 test_util.cc:175] -----------------------------------------------
I20260811 11:50:33.773000 3957 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786449004114841-3957-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (347 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260811 11:50:33.774184 3957 mini_oidc.cc:152] Starting JWKS server
I20260811 11:50:33.775081 3957 webserver.cc:460] Webserver started at https://127.0.0.1:45883/ using document root <none> and password file <none>
I20260811 11:50:33.775182 3957 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 11:50:33.775439 3957 webserver.cc:460] Webserver started at http://127.0.0.1:35401/ using document root <none> and password file <none>
2026-08-11T11:50:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:50:33Z Disabled control of system clock
I20260811 11:50:33.785694 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:45523
--webserver_interface=127.3.221.126
--webserver_port=0
--builtin_ntp_servers=127.3.221.84:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:45523
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45883/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786449004114841-3957-0/testchainca.cert with env {}
W20260811 11:50:33.902493 12638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:33.902781 12638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:33.902820 12638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:33.906847 12638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:50:33.906909 12638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:33.906926 12638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:33.906948 12638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:50:33.906970 12638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 11:50:33.907002 12638 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 11:50:33.907027 12638 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45883/jwks.json
I20260811 11:50:33.912524 12638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:45523
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:45523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45883/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786449004114841-3957-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:33.913667 12638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:33.914685 12638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:33.923321 12645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:33.923336 12646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:33.923321 12648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:33.923693 12638 server_base.cc:1034] running on GCE node
I20260811 11:50:33.924297 12638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:33.925529 12638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:33.926702 12638 hybrid_clock.cc:648] HybridClock initialized: now 1786449033926682 us; error 32 us; skew 500 ppm
I20260811 11:50:33.928568 12638 webserver.cc:460] Webserver started at http://127.3.221.126:46591/ using document root <none> and password file <none>
I20260811 11:50:33.929157 12638 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:33.929253 12638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:33.929476 12638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:33.931569 12638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "384615890c11400d8d2e62a349a36795"
format_stamp: "Formatted at 2026-08-11 11:50:33 on dist-test-slave-bndk"
server_key: "c9aaeda979bf2a580650efd6f2565a43"
server_key_iv: "5183703e17cca736371cf392ca248891"
server_key_version: "encryptionkey@0"
I20260811 11:50:33.932129 12638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "384615890c11400d8d2e62a349a36795"
format_stamp: "Formatted at 2026-08-11 11:50:33 on dist-test-slave-bndk"
server_key: "c9aaeda979bf2a580650efd6f2565a43"
server_key_iv: "5183703e17cca736371cf392ca248891"
server_key_version: "encryptionkey@0"
I20260811 11:50:33.935833 12638 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260811 11:50:33.938344 12656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:33.939579 12638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 11:50:33.939702 12638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "384615890c11400d8d2e62a349a36795"
format_stamp: "Formatted at 2026-08-11 11:50:33 on dist-test-slave-bndk"
server_key: "c9aaeda979bf2a580650efd6f2565a43"
server_key_iv: "5183703e17cca736371cf392ca248891"
server_key_version: "encryptionkey@0"
I20260811 11:50:33.939807 12638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:33.957893 12638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:33.958743 12638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:33.958942 12638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:33.966370 12638 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:45523
I20260811 11:50:33.966384 12746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:45523 every 8 connection(s)
I20260811 11:50:33.967397 12638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:50:33.970810 12749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:33.972431 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 12638
I20260811 11:50:33.972550 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:50:33.972841 3957 external_mini_cluster.cc:1428] Setting key e380c783539500722c7ac5fcd87c7069
I20260811 11:50:33.977631 12749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795: Bootstrap starting.
I20260811 11:50:33.980020 12749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:33.980775 12749 log.cc:824] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:33.982429 12754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:33.974347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34234 (local address 127.3.221.126:45523)
0811 11:50:33.975168 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:33.975187 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:33.975227 (+ 40us) server_negotiation.cc:407] Connection header received
0811 11:50:33.975995 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:33.976014 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:33.976434 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:33.976750 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:33.977198 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:33.978258 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:33.979148 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:33.979636 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:33.979798 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:33.980634 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:33.980639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:33.980654 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:50:33.981088 (+ 434us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:33.981150 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:50:33.981171 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:33.981288 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:33.981426 (+ 138us) server_negotiation.cc:299] Negotiation successful
0811 11:50:33.981689 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":152,"threads_started":1}
I20260811 11:50:33.982831 12749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795: No bootstrap required, opened a new log
I20260811 11:50:33.985303 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:50:33.986963 12749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384615890c11400d8d2e62a349a36795" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 45523 } }
I20260811 11:50:33.987196 12749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:33.987267 12749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384615890c11400d8d2e62a349a36795, State: Initialized, Role: FOLLOWER
I20260811 11:50:33.987689 12749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384615890c11400d8d2e62a349a36795" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 45523 } }
I20260811 11:50:33.987819 12749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:50:33.987885 12749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:50:33.987986 12749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:33.988891 12749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384615890c11400d8d2e62a349a36795" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 45523 } }
I20260811 11:50:33.989241 12749 leader_election.cc:302] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 384615890c11400d8d2e62a349a36795; no voters:
I20260811 11:50:33.989599 12749 leader_election.cc:288] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:50:33.989733 12757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:33.989991 12757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795 [term 1 LEADER]: Becoming Leader. State: Replica: 384615890c11400d8d2e62a349a36795, State: Running, Role: LEADER
I20260811 11:50:33.990336 12757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384615890c11400d8d2e62a349a36795" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 45523 } }
I20260811 11:50:33.990839 12749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:50:33.992010 12759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 384615890c11400d8d2e62a349a36795. Latest consensus state: current_term: 1 leader_uuid: "384615890c11400d8d2e62a349a36795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384615890c11400d8d2e62a349a36795" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 45523 } } }
I20260811 11:50:33.992161 12759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:33.992544 12771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:50:33.992913 12758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "384615890c11400d8d2e62a349a36795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384615890c11400d8d2e62a349a36795" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 45523 } } }
I20260811 11:50:33.993028 12758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:33.996011 12771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:50:34.001794 12771 catalog_manager.cc:1347] Generated new cluster ID: 0855fda329ab4d6d857b45842595677b
I20260811 11:50:34.001878 12771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:50:34.009630 12771 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:50:34.010584 12771 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:50:34.019549 12771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 384615890c11400d8d2e62a349a36795: Generated new TSK 0
I20260811 11:50:34.020468 12771 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260811 11:50:34.048606 12801 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260811 11:50:34.048714 12801 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260811 11:50:34.049423 12754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:34.033396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34246 (local address 127.3.221.126:45523)
0811 11:50:34.033528 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:34.033533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:34.033677 (+ 144us) server_negotiation.cc:407] Connection header received
0811 11:50:34.033841 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:34.033844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:34.033897 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:34.033991 (+ 94us) server_negotiation.cc:226] Negotiated authn=JWT
0811 11:50:34.034231 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:34.034805 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:34.036011 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:34.036224 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:34.036295 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:34.036606 (+ 311us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 11:50:34.049010 (+ 12404us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45883/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0811 11:50:34.049291 (+ 281us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34246: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45883/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":55}
W20260811 11:50:34.049491 12754 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34246: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45883/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260811 11:50:34.049678 12800 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:34.033263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:45523 (local address 127.0.0.1:34246)
0811 11:50:34.033585 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:34.033600 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:34.033720 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:34.034005 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:34.034008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:34.034016 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0811 11:50:34.034139 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:34.034146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:34.034912 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:34.034916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:34.035902 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:34.035914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:34.036333 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:34.036337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:34.036424 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:34.036455 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 11:50:34.049256 (+ 12801us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45883/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0811 11:50:34.049473 (+ 217us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.221.126:45523: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45883/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":117,"threads_started":1}
I20260811 11:50:34.050491 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 12638
2026-08-11T11:50:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (328 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260811 11:50:34.102579 3957 mini_oidc.cc:152] Starting JWKS server
I20260811 11:50:34.103482 3957 webserver.cc:460] Webserver started at https://127.0.0.1:37081/ using document root <none> and password file <none>
I20260811 11:50:34.103578 3957 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 11:50:34.103830 3957 webserver.cc:460] Webserver started at http://127.0.0.1:40353/ using document root <none> and password file <none>
2026-08-11T11:50:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:50:34Z Disabled control of system clock
I20260811 11:50:34.114562 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:39349
--webserver_interface=127.3.221.126
--webserver_port=0
--builtin_ntp_servers=127.3.221.84:33417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:39349
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37081/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260811 11:50:34.232386 12813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:34.232700 12813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:34.232759 12813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:34.237164 12813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:50:34.237296 12813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:34.237322 12813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:34.237344 12813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:50:34.237363 12813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 11:50:34.237387 12813 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 11:50:34.237413 12813 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37081/jwks.json
I20260811 11:50:34.242623 12813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:33417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:39349
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:39349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37081/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:34.243839 12813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:34.245555 12813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:34.254590 12818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:34.254733 12813 server_base.cc:1034] running on GCE node
W20260811 11:50:34.254603 12821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:34.254588 12819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:34.255554 12813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:34.256788 12813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:34.257989 12813 hybrid_clock.cc:648] HybridClock initialized: now 1786449034257962 us; error 45 us; skew 500 ppm
I20260811 11:50:34.260099 12813 webserver.cc:460] Webserver started at http://127.3.221.126:35903/ using document root <none> and password file <none>
I20260811 11:50:34.260773 12813 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:34.260852 12813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:34.261073 12813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:34.263396 12813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "851f581c09664b6faf173090a95a4212"
format_stamp: "Formatted at 2026-08-11 11:50:34 on dist-test-slave-bndk"
server_key: "fa66fdc6258c7eb9127f59c5d556a5d6"
server_key_iv: "a50daec19ab6ce11142b2364c1d0773f"
server_key_version: "encryptionkey@0"
I20260811 11:50:34.264002 12813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "851f581c09664b6faf173090a95a4212"
format_stamp: "Formatted at 2026-08-11 11:50:34 on dist-test-slave-bndk"
server_key: "fa66fdc6258c7eb9127f59c5d556a5d6"
server_key_iv: "a50daec19ab6ce11142b2364c1d0773f"
server_key_version: "encryptionkey@0"
I20260811 11:50:34.268170 12813 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260811 11:50:34.270963 12828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:34.272511 12813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 11:50:34.272671 12813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "851f581c09664b6faf173090a95a4212"
format_stamp: "Formatted at 2026-08-11 11:50:34 on dist-test-slave-bndk"
server_key: "fa66fdc6258c7eb9127f59c5d556a5d6"
server_key_iv: "a50daec19ab6ce11142b2364c1d0773f"
server_key_version: "encryptionkey@0"
I20260811 11:50:34.272802 12813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:34.287782 12813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:34.288794 12813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:34.289021 12813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:34.297250 12813 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:39349
I20260811 11:50:34.297325 12913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:39349 every 8 connection(s)
I20260811 11:50:34.298341 12813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:50:34.301482 12919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:34.301635 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 12813
I20260811 11:50:34.301748 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:50:34.302032 3957 external_mini_cluster.cc:1428] Setting key d04cd7ec0fa65493385573efff7c8ffc
I20260811 11:50:34.309638 12919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212: Bootstrap starting.
I20260811 11:50:34.312410 12919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:34.313076 12925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:34.303478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.3.221.126:39349)
0811 11:50:34.304354 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:34.304376 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:34.304418 (+ 42us) server_negotiation.cc:407] Connection header received
0811 11:50:34.305362 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:34.305393 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:34.305950 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:34.306379 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:34.306916 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:34.308064 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:34.309323 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:34.309867 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:34.309992 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:34.310902 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:34.310908 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:34.310924 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:50:34.311589 (+ 665us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:34.311666 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:50:34.311692 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:34.311785 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:34.311946 (+ 161us) server_negotiation.cc:299] Negotiation successful
0811 11:50:34.312255 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":148,"threads_started":1}
I20260811 11:50:34.313261 12919 log.cc:824] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:34.315438 12919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212: No bootstrap required, opened a new log
I20260811 11:50:34.316143 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:50:34.319476 12919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851f581c09664b6faf173090a95a4212" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 39349 } }
I20260811 11:50:34.319733 12919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:34.319768 12919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 851f581c09664b6faf173090a95a4212, State: Initialized, Role: FOLLOWER
I20260811 11:50:34.320163 12919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851f581c09664b6faf173090a95a4212" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 39349 } }
I20260811 11:50:34.320286 12919 raft_consensus.cc:388] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:50:34.320333 12919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:50:34.320410 12919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:34.321426 12919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851f581c09664b6faf173090a95a4212" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 39349 } }
I20260811 11:50:34.321766 12919 leader_election.cc:302] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 851f581c09664b6faf173090a95a4212; no voters:
I20260811 11:50:34.322069 12919 leader_election.cc:288] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:50:34.322199 12929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:34.322443 12929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212 [term 1 LEADER]: Becoming Leader. State: Replica: 851f581c09664b6faf173090a95a4212, State: Running, Role: LEADER
I20260811 11:50:34.322806 12929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851f581c09664b6faf173090a95a4212" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 39349 } }
I20260811 11:50:34.323451 12919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:50:34.324712 12932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 851f581c09664b6faf173090a95a4212. Latest consensus state: current_term: 1 leader_uuid: "851f581c09664b6faf173090a95a4212" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851f581c09664b6faf173090a95a4212" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 39349 } } }
I20260811 11:50:34.324760 12931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "851f581c09664b6faf173090a95a4212" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851f581c09664b6faf173090a95a4212" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 39349 } } }
I20260811 11:50:34.324918 12931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:34.324927 12932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:34.325520 12945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:50:34.328362 12945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:50:34.334080 12945 catalog_manager.cc:1347] Generated new cluster ID: 4c6313b5496448c8a661a345c193da43
I20260811 11:50:34.334167 12945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:50:34.350212 12945 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:50:34.351200 12945 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:50:34.356422 12945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 851f581c09664b6faf173090a95a4212: Generated new TSK 0
I20260811 11:50:34.357152 12945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:50:34.380147 12925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:34.373871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.3.221.126:39349)
0811 11:50:34.374013 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:34.374018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:34.374175 (+ 157us) server_negotiation.cc:407] Connection header received
0811 11:50:34.374356 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:34.374360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:34.374426 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:34.374564 (+ 138us) server_negotiation.cc:226] Negotiated authn=JWT
0811 11:50:34.374852 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:34.375682 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:34.377269 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:34.377545 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:34.377636 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:34.378144 (+ 508us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 11:50:34.379725 (+ 1581us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37081/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0811 11:50:34.379995 (+ 270us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35228: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37081/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":49}
W20260811 11:50:34.380234 12925 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35228: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37081/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260811 11:50:34.381829 12966 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:34.373748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:39349 (local address 127.0.0.1:35228)
0811 11:50:34.374088 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:34.374101 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:34.374214 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:34.374611 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:34.374615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:34.374623 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0811 11:50:34.374739 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:34.374745 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:34.375966 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:34.375971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:34.376893 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:34.376913 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:34.377810 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:34.377814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:34.377922 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:34.377948 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 11:50:34.379955 (+ 2007us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37081/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0811 11:50:34.381638 (+ 1683us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.221.126:39349: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37081/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":1444,"thread_start_us":133,"threads_started":1}
I20260811 11:50:34.382226 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 12813
E20260811 11:50:34.406567 12968 webserver.cc:533] Webserver: sslize failed:
E20260811 11:50:34.406664 12968 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-11T11:50:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (334 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-11T11:50:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:50:34Z Disabled control of system clock
I20260811 11:50:34.447453 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:46669
--webserver_interface=127.3.221.126
--webserver_port=0
--builtin_ntp_servers=127.3.221.84:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:46669
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786449004114841-3957-0/testchainca.cert with env {}
W20260811 11:50:34.563311 12972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:34.563603 12972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:34.563644 12972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:34.567723 12972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:50:34.567795 12972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:34.567809 12972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:34.567826 12972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:50:34.567838 12972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 11:50:34.567853 12972 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 11:50:34.567865 12972 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260811 11:50:34.573509 12972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:46669
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:46669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786449004114841-3957-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:34.574654 12972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:34.575700 12972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:34.584909 12986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:34.584914 12983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:34.584914 12981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:34.584975 12972 server_base.cc:1034] running on GCE node
I20260811 11:50:34.585876 12972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:34.587098 12972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:34.588267 12972 hybrid_clock.cc:648] HybridClock initialized: now 1786449034588255 us; error 33 us; skew 500 ppm
I20260811 11:50:34.590384 12972 webserver.cc:460] Webserver started at http://127.3.221.126:39989/ using document root <none> and password file <none>
I20260811 11:50:34.591009 12972 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:34.591079 12972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:34.591316 12972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:34.593611 12972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "fbf6b48de3294f7a935ed2a84325229f"
format_stamp: "Formatted at 2026-08-11 11:50:34 on dist-test-slave-bndk"
server_key: "d38d6bbc3d18129ef3fb04d9bbd2a579"
server_key_iv: "80d53af143930c5ea8fba9f9d4005301"
server_key_version: "encryptionkey@0"
I20260811 11:50:34.594226 12972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "fbf6b48de3294f7a935ed2a84325229f"
format_stamp: "Formatted at 2026-08-11 11:50:34 on dist-test-slave-bndk"
server_key: "d38d6bbc3d18129ef3fb04d9bbd2a579"
server_key_iv: "80d53af143930c5ea8fba9f9d4005301"
server_key_version: "encryptionkey@0"
I20260811 11:50:34.598284 12972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 11:50:34.600946 12997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:34.602416 12972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 11:50:34.602553 12972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "fbf6b48de3294f7a935ed2a84325229f"
format_stamp: "Formatted at 2026-08-11 11:50:34 on dist-test-slave-bndk"
server_key: "d38d6bbc3d18129ef3fb04d9bbd2a579"
server_key_iv: "80d53af143930c5ea8fba9f9d4005301"
server_key_version: "encryptionkey@0"
I20260811 11:50:34.602676 12972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:34.622475 12972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:34.623239 12972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:34.623441 12972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:34.630491 12972 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:46669
I20260811 11:50:34.630522 13079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:46669 every 8 connection(s)
I20260811 11:50:34.631541 12972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:50:34.634285 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 12972
I20260811 11:50:34.634418 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:50:34.634696 3957 external_mini_cluster.cc:1428] Setting key f9a74196173238b4d9d12ef391f88f53
I20260811 11:50:34.635061 13080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:34.642742 13080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f: Bootstrap starting.
I20260811 11:50:34.645720 13080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:34.645761 13090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:34.636139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51390 (local address 127.3.221.126:46669)
0811 11:50:34.636870 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:34.636905 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:34.636949 (+ 44us) server_negotiation.cc:407] Connection header received
0811 11:50:34.637929 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:34.637954 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:34.638515 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:34.638899 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:34.639370 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:34.640793 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:34.641846 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:34.642550 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:34.642718 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:34.643595 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:34.643602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:34.643618 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:50:34.644131 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:34.644211 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:50:34.644240 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:34.644350 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:34.644445 (+ 95us) server_negotiation.cc:299] Negotiation successful
0811 11:50:34.644724 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":114,"threads_started":1}
I20260811 11:50:34.646575 13080 log.cc:824] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:34.648739 13080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f: No bootstrap required, opened a new log
I20260811 11:50:34.648734 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:50:34.652567 13080 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf6b48de3294f7a935ed2a84325229f" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46669 } }
I20260811 11:50:34.652850 13080 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:34.652915 13080 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbf6b48de3294f7a935ed2a84325229f, State: Initialized, Role: FOLLOWER
I20260811 11:50:34.653373 13080 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf6b48de3294f7a935ed2a84325229f" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46669 } }
I20260811 11:50:34.653518 13080 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:50:34.653584 13080 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:50:34.653693 13080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:34.654831 13080 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf6b48de3294f7a935ed2a84325229f" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46669 } }
I20260811 11:50:34.655154 13080 leader_election.cc:302] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbf6b48de3294f7a935ed2a84325229f; no voters:
I20260811 11:50:34.655434 13080 leader_election.cc:288] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:50:34.655577 13095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:34.655782 13095 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f [term 1 LEADER]: Becoming Leader. State: Replica: fbf6b48de3294f7a935ed2a84325229f, State: Running, Role: LEADER
I20260811 11:50:34.656105 13095 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf6b48de3294f7a935ed2a84325229f" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46669 } }
I20260811 11:50:34.656670 13080 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:50:34.657790 13096 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbf6b48de3294f7a935ed2a84325229f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf6b48de3294f7a935ed2a84325229f" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46669 } } }
I20260811 11:50:34.657932 13096 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:34.658293 13114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:50:34.658950 13098 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbf6b48de3294f7a935ed2a84325229f. Latest consensus state: current_term: 1 leader_uuid: "fbf6b48de3294f7a935ed2a84325229f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf6b48de3294f7a935ed2a84325229f" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46669 } } }
I20260811 11:50:34.659066 13098 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:34.661861 13114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:50:34.667999 13114 catalog_manager.cc:1347] Generated new cluster ID: 44fe5b8a625b4e32995194a81386e704
I20260811 11:50:34.668071 13114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:50:34.688494 13114 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:50:34.689990 13114 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:50:34.696039 13114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbf6b48de3294f7a935ed2a84325229f: Generated new TSK 0
I20260811 11:50:34.696920 13114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:50:35.594426 13090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:34.705542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.3.221.126:46669)
0811 11:50:34.705685 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:34.705690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:34.705877 (+ 187us) server_negotiation.cc:407] Connection header received
0811 11:50:34.706045 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:34.706047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:34.706097 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:34.706186 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0811 11:50:34.706426 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:34.707032 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:34.708288 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:34.708498 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:34.708575 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:34.708941 (+ 366us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 11:50:35.593865 (+884924us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0811 11:50:35.594227 (+ 362us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51406: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":55}
W20260811 11:50:35.594527 13090 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51406: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260811 11:50:35.595094 13140 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:34.705468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46669 (local address 127.0.0.1:51406)
0811 11:50:34.705788 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:34.705803 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:34.705927 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:34.706199 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:34.706202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:34.706209 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0811 11:50:34.706325 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:34.706332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:34.707142 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:34.707146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:34.708173 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:34.708185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:34.708613 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:34.708617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:34.708714 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:34.708745 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 11:50:35.594721 (+885976us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0811 11:50:35.594936 (+ 215us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.221.126:46669: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":94,"threads_started":1}
I20260811 11:50:35.595674 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 12972
2026-08-11T11:50:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1188 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (132 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (131 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:50:35 dist-test-slave-bndk krb5kdc[13156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:50:35 dist-test-slave-bndk krb5kdc[13156](info): set up 2 sockets
Aug 11 11:50:35 dist-test-slave-bndk krb5kdc[13156](info): commencing operation
krb5kdc: starting...
W20260811 11:50:37.034278 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:50:37 dist-test-slave-bndk krb5kdc[13156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:50:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:50:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:50:37.170406 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:32993
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:38699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:32993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:37.288575 13173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:37.288908 13173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:37.288971 13173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:37.293200 13173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:50:37.293323 13173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:37.293352 13173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:37.293375 13173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:50:37.293394 13173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:50:37.299093 13173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:38699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:32993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:32993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:37.300570 13173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:37.301728 13173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:37.310771 13187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:37.310776 13184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:37.310771 13182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:37.310992 13173 server_base.cc:1034] running on GCE node
I20260811 11:50:37.311741 13173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:37.312963 13173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:37.314182 13173 hybrid_clock.cc:648] HybridClock initialized: now 1786449037314160 us; error 55 us; skew 500 ppm
Aug 11 11:50:37 dist-test-slave-bndk krb5kdc[13156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449037, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:37.320596 13173 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:50:37.321744 13173 webserver.cc:460] Webserver started at http://127.3.221.126:36903/ using document root <none> and password file <none>
I20260811 11:50:37.322350 13173 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:37.322422 13173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:37.322642 13173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:37.324908 13173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "8ed6fd8a95324044b32d05c7999a0f88"
format_stamp: "Formatted at 2026-08-11 11:50:37 on dist-test-slave-bndk"
server_key: "7847fd91a1a9cec80fc64f45f5a3017f"
server_key_iv: "71508f05539b1db3b85a5b974412914b"
server_key_version: "encryptionkey@0"
I20260811 11:50:37.325562 13173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "8ed6fd8a95324044b32d05c7999a0f88"
format_stamp: "Formatted at 2026-08-11 11:50:37 on dist-test-slave-bndk"
server_key: "7847fd91a1a9cec80fc64f45f5a3017f"
server_key_iv: "71508f05539b1db3b85a5b974412914b"
server_key_version: "encryptionkey@0"
I20260811 11:50:37.329646 13173 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:50:37.332515 13198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:37.333963 13173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:50:37.334102 13173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "8ed6fd8a95324044b32d05c7999a0f88"
format_stamp: "Formatted at 2026-08-11 11:50:37 on dist-test-slave-bndk"
server_key: "7847fd91a1a9cec80fc64f45f5a3017f"
server_key_iv: "71508f05539b1db3b85a5b974412914b"
server_key_version: "encryptionkey@0"
I20260811 11:50:37.334218 13173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:37.370819 13173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:37.374119 13173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:37.374322 13173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:37.381680 13173 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:32993
I20260811 11:50:37.381709 13280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:32993 every 8 connection(s)
I20260811 11:50:37.382747 13173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:50:37.385797 13281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:37.388760 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 13173
I20260811 11:50:37.388895 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:50:37.389154 3957 external_mini_cluster.cc:1428] Setting key 526dd7bb8b83e4e225ec656fdf892b55
I20260811 11:50:37.392479 13281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88: Bootstrap starting.
I20260811 11:50:37.395277 13281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:37.396236 13281 log.cc:824] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:37.398485 13281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88: No bootstrap required, opened a new log
Aug 11 11:50:37 dist-test-slave-bndk krb5kdc[13156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:37.403355 13281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed6fd8a95324044b32d05c7999a0f88" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 32993 } }
I20260811 11:50:37.403694 13281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:37.403764 13281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed6fd8a95324044b32d05c7999a0f88, State: Initialized, Role: FOLLOWER
I20260811 11:50:37.404309 13281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed6fd8a95324044b32d05c7999a0f88" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 32993 } }
I20260811 11:50:37.404481 13281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:50:37.404551 13281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:50:37.404649 13281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:37.404912 13287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:37.390677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.3.221.126:32993)
0811 11:50:37.391264 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:37.391281 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:37.391346 (+ 65us) server_negotiation.cc:407] Connection header received
0811 11:50:37.392019 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:37.392037 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:37.392417 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:37.392761 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:37.393724 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:37.394777 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:37.395909 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:37.396488 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:37.396627 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:37.399197 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:37.399204 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:37.399223 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:37.399259 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:37.401422 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:37.401892 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:37.401898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:37.401905 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:37.402072 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:37.402335 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:37.402339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:37.402341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:37.402980 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:37.403207 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:37.403475 (+ 268us) server_negotiation.cc:299] Negotiation successful
0811 11:50:37.403834 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":169,"threads_started":1}
I20260811 11:50:37.405841 13281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed6fd8a95324044b32d05c7999a0f88" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 32993 } }
I20260811 11:50:37.406203 13281 leader_election.cc:302] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ed6fd8a95324044b32d05c7999a0f88; no voters:
I20260811 11:50:37.406571 13281 leader_election.cc:288] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:50:37.406697 13289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:37.406936 13289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88 [term 1 LEADER]: Becoming Leader. State: Replica: 8ed6fd8a95324044b32d05c7999a0f88, State: Running, Role: LEADER
I20260811 11:50:37.407320 13289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed6fd8a95324044b32d05c7999a0f88" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 32993 } }
I20260811 11:50:37.407883 13281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:50:37.408618 13289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ed6fd8a95324044b32d05c7999a0f88. Latest consensus state: current_term: 1 leader_uuid: "8ed6fd8a95324044b32d05c7999a0f88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed6fd8a95324044b32d05c7999a0f88" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 32993 } } }
I20260811 11:50:37.408736 13289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:37.409068 13291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ed6fd8a95324044b32d05c7999a0f88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed6fd8a95324044b32d05c7999a0f88" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 32993 } } }
I20260811 11:50:37.409267 13291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:37.409279 13300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:50:37.412750 13300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:50:37.419332 13300 catalog_manager.cc:1347] Generated new cluster ID: 4fca5dcd39f44b6b8e55d8d83957b5b6
I20260811 11:50:37.419432 13300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:50:37.432253 13300 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:50:37.433334 13300 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:50:37.438047 13300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ed6fd8a95324044b32d05c7999a0f88: Generated new TSK 0
I20260811 11:50:37.439019 13300 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
I20260811 11:50:37.481365 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:32993
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:38699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:37.602607 13319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:37.602909 13319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:37.602954 13319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:37.607156 13319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:37.607249 13319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:37.607314 13319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:50:37.613142 13319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:38699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:32993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:37.614395 13319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:37.615703 13319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:37.625802 13330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:37.625813 13326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:37.625808 13325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:37.626983 13319 server_base.cc:1034] running on GCE node
I20260811 11:50:37.627424 13319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:37.628082 13319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:37.629315 13319 hybrid_clock.cc:648] HybridClock initialized: now 1786449037629290 us; error 34 us; skew 500 ppm
Aug 11 11:50:37 dist-test-slave-bndk krb5kdc[13156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449037, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:37.632167 13319 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:50:37.633167 13319 webserver.cc:460] Webserver started at http://127.3.221.65:38739/ using document root <none> and password file <none>
I20260811 11:50:37.633824 13319 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:37.633898 13319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:37.634078 13319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:37.636233 13319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "ef9f871e15634bbcaf54d835e094cfc8"
format_stamp: "Formatted at 2026-08-11 11:50:37 on dist-test-slave-bndk"
server_key: "835514db49db37a15b50e526f4898c51"
server_key_iv: "d63bbecc99f8020cb7e8af431c602cb6"
server_key_version: "encryptionkey@0"
I20260811 11:50:37.636807 13319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "ef9f871e15634bbcaf54d835e094cfc8"
format_stamp: "Formatted at 2026-08-11 11:50:37 on dist-test-slave-bndk"
server_key: "835514db49db37a15b50e526f4898c51"
server_key_iv: "d63bbecc99f8020cb7e8af431c602cb6"
server_key_version: "encryptionkey@0"
I20260811 11:50:37.640954 13319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260811 11:50:37.643767 13343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:37.645121 13319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260811 11:50:37.645272 13319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "ef9f871e15634bbcaf54d835e094cfc8"
format_stamp: "Formatted at 2026-08-11 11:50:37 on dist-test-slave-bndk"
server_key: "835514db49db37a15b50e526f4898c51"
server_key_iv: "d63bbecc99f8020cb7e8af431c602cb6"
server_key_version: "encryptionkey@0"
I20260811 11:50:37.645408 13319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:37.657716 13319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:37.661046 13319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:37.661266 13319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:37.661896 13319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:37.662789 13319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:37.662846 13319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:37.662889 13319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:37.662904 13319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:37.671780 13319 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:43459
I20260811 11:50:37.671808 13524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:43459 every 8 connection(s)
I20260811 11:50:37.672752 13319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:50:37.679873 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 13319
I20260811 11:50:37.680044 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:50:37.680361 3957 external_mini_cluster.cc:1428] Setting key a97f3ef163f11d8b717acf0cdea3a67b
Aug 11 11:50:37 dist-test-slave-bndk krb5kdc[13156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449037, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:37.687855 13287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:37.674813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:55833 (local address 127.3.221.126:32993)
0811 11:50:37.674999 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:37.675004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:37.675893 (+ 889us) server_negotiation.cc:407] Connection header received
0811 11:50:37.677311 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:37.677313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:37.677367 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:37.677443 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:37.678719 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:37.679281 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:37.680919 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:37.681200 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:37.681344 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:37.684187 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:37.684191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:37.684193 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:37.684223 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:37.685648 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:37.686188 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:37.686191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:37.686192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:37.686283 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:37.686658 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:37.686662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:37.686664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:37.686860 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:37.686942 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:37.687532 (+ 590us) server_negotiation.cc:299] Negotiation successful
0811 11:50:37.687680 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260811 11:50:37.688733 13528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:37.675217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:32993 (local address 127.3.221.65:55833)
0811 11:50:37.675725 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:37.675773 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:37.677054 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:37.677616 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:37.677622 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:37.678020 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:37.678558 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:37.678574 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:37.679427 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:37.679430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:37.680779 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:37.680791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:37.681383 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:37.681386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:37.681554 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:37.682264 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:37.682283 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:37.683999 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:37.685779 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:37.685786 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:37.685806 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:37.686085 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:37.686379 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:37.686383 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:37.686385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:37.686557 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:37.686974 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:37.686981 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:37.687260 (+ 279us) client_negotiation.cc:769] Sending connection context
0811 11:50:37.687451 (+ 191us) client_negotiation.cc:240] Negotiation successful
0811 11:50:37.687847 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":131,"threads_started":1}
I20260811 11:50:37.690204 13526 heartbeater.cc:347] Connected to a master server at 127.3.221.126:32993
I20260811 11:50:37.690490 13526 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.221.66@KRBTEST.COM; defaulting to no policy
I20260811 11:50:37.691152 13526 heartbeater.cc:511] Master 127.3.221.126:32993 requested a full tablet report, sending...
I20260811 11:50:37.693138 13223 ts_manager.cc:194] Registered new tserver with Master: ef9f871e15634bbcaf54d835e094cfc8 (127.3.221.65:43459)
I20260811 11:50:37.694775 13223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:55833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
I20260811 11:50:37.712405 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:32993
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:38699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:37.830006 13532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:37.830279 13532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:37.830340 13532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:37.834445 13532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:37.834522 13532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:37.834583 13532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:50:37.840030 13532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:38699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:32993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:37.841105 13532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:37.842160 13532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:37.852551 13547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:37.852566 13545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:37.852661 13532 server_base.cc:1034] running on GCE node
W20260811 11:50:37.852566 13543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:37.853250 13532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:37.853930 13532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:37.855118 13532 hybrid_clock.cc:648] HybridClock initialized: now 1786449037855089 us; error 36 us; skew 500 ppm
Aug 11 11:50:37 dist-test-slave-bndk krb5kdc[13156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449037, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:37.858035 13532 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:50:37.859045 13532 webserver.cc:460] Webserver started at http://127.3.221.66:44651/ using document root <none> and password file <none>
I20260811 11:50:37.859690 13532 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:37.859768 13532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:37.860001 13532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:37.862239 13532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "cdb2488a024a4c3198fbafc12940a299"
format_stamp: "Formatted at 2026-08-11 11:50:37 on dist-test-slave-bndk"
server_key: "a79c4ae7b957be00ea046de22a9ef998"
server_key_iv: "78d90fb324db68c9b47a8a96a6183f8e"
server_key_version: "encryptionkey@0"
I20260811 11:50:37.862818 13532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "cdb2488a024a4c3198fbafc12940a299"
format_stamp: "Formatted at 2026-08-11 11:50:37 on dist-test-slave-bndk"
server_key: "a79c4ae7b957be00ea046de22a9ef998"
server_key_iv: "78d90fb324db68c9b47a8a96a6183f8e"
server_key_version: "encryptionkey@0"
I20260811 11:50:37.866901 13532 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:50:37.869537 13558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:37.870836 13532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 11:50:37.870958 13532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "cdb2488a024a4c3198fbafc12940a299"
format_stamp: "Formatted at 2026-08-11 11:50:37 on dist-test-slave-bndk"
server_key: "a79c4ae7b957be00ea046de22a9ef998"
server_key_iv: "78d90fb324db68c9b47a8a96a6183f8e"
server_key_version: "encryptionkey@0"
I20260811 11:50:37.871083 13532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:37.897722 13532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:37.901027 13532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:37.901211 13532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:37.901896 13532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:37.902770 13532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:37.902817 13532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:37.902859 13532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:37.902889 13532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:37.912124 13532 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:33645
I20260811 11:50:37.912150 13724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:33645 every 8 connection(s)
I20260811 11:50:37.913077 13532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:50:37.920594 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 13532
I20260811 11:50:37.920723 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:50:37.920997 3957 external_mini_cluster.cc:1428] Setting key 8db660cd937d942ac02e47c800b4d3b2
Aug 11 11:50:37 dist-test-slave-bndk krb5kdc[13156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449037, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:37.927438 13287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:37.914943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:39003 (local address 127.3.221.126:32993)
0811 11:50:37.915074 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:37.915078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:37.916027 (+ 949us) server_negotiation.cc:407] Connection header received
0811 11:50:37.917449 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:37.917453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:37.917515 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:37.917595 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:37.918830 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:37.919360 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:37.920527 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:37.920757 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:37.920857 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:37.923668 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:37.923672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:37.923675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:37.923708 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:37.925248 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:37.925765 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:37.925769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:37.925771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:37.925877 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:37.926238 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:37.926241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:37.926243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:37.926467 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:37.926565 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:37.927139 (+ 574us) server_negotiation.cc:299] Negotiation successful
0811 11:50:37.927270 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 11:50:37.928332 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:37.915396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:32993 (local address 127.3.221.66:39003)
0811 11:50:37.915873 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:37.915921 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:37.917193 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:37.917789 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:37.917796 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:37.918177 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:37.918679 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:37.918693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:37.919488 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:37.919491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:37.920394 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:37.920404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:37.920908 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:37.920912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:37.921091 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:37.921705 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:37.921729 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:37.923468 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:37.925364 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:37.925370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:37.925383 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:37.925665 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:37.925978 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:37.925981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:37.925982 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:37.926146 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:37.926599 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:37.926605 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:37.926888 (+ 283us) client_negotiation.cc:769] Sending connection context
0811 11:50:37.927062 (+ 174us) client_negotiation.cc:240] Negotiation successful
0811 11:50:37.927345 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":125,"threads_started":1}
I20260811 11:50:37.929584 13725 heartbeater.cc:347] Connected to a master server at 127.3.221.126:32993
I20260811 11:50:37.929898 13725 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:37.930549 13725 heartbeater.cc:511] Master 127.3.221.126:32993 requested a full tablet report, sending...
I20260811 11:50:37.931907 13223 ts_manager.cc:194] Registered new tserver with Master: cdb2488a024a4c3198fbafc12940a299 (127.3.221.66:33645)
WARNING: no policy specified for kudu/127.3.221.67@KRBTEST.COM; defaulting to no policy
I20260811 11:50:37.932548 13223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:39003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
I20260811 11:50:37.952724 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:32993
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:38699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:38.068562 13736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:38.068859 13736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:38.068903 13736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:38.073073 13736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:38.073160 13736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:38.073242 13736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:50:38.079070 13736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:38699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:32993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:38.080283 13736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:38.081419 13736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:38.091703 13742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:38.091714 13741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:38.091699 13745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:38.092046 13736 server_base.cc:1034] running on GCE node
I20260811 11:50:38.092554 13736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:38.093253 13736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:38.094462 13736 hybrid_clock.cc:648] HybridClock initialized: now 1786449038094427 us; error 53 us; skew 500 ppm
Aug 11 11:50:38 dist-test-slave-bndk krb5kdc[13156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449038, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:38.097266 13736 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:50:38.098212 13736 webserver.cc:460] Webserver started at http://127.3.221.67:35479/ using document root <none> and password file <none>
I20260811 11:50:38.098795 13736 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:38.098865 13736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:38.099068 13736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:38.101287 13736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "8f38279f5ff048309df43e7568a5b890"
format_stamp: "Formatted at 2026-08-11 11:50:38 on dist-test-slave-bndk"
server_key: "e164e6acca90598f58c676949836a020"
server_key_iv: "4afaf22a400f21da859ffafa46c00a0e"
server_key_version: "encryptionkey@0"
I20260811 11:50:38.101862 13736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "8f38279f5ff048309df43e7568a5b890"
format_stamp: "Formatted at 2026-08-11 11:50:38 on dist-test-slave-bndk"
server_key: "e164e6acca90598f58c676949836a020"
server_key_iv: "4afaf22a400f21da859ffafa46c00a0e"
server_key_version: "encryptionkey@0"
I20260811 11:50:38.105751 13736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 11:50:38.108374 13755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:38.109871 13736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:50:38.109994 13736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "8f38279f5ff048309df43e7568a5b890"
format_stamp: "Formatted at 2026-08-11 11:50:38 on dist-test-slave-bndk"
server_key: "e164e6acca90598f58c676949836a020"
server_key_iv: "4afaf22a400f21da859ffafa46c00a0e"
server_key_version: "encryptionkey@0"
I20260811 11:50:38.110100 13736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:38.131572 13736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:38.134567 13736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:38.134780 13736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:38.135593 13736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:38.136559 13736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:38.136610 13736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:38.136649 13736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:38.136669 13736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:38.145753 13736 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:39745
I20260811 11:50:38.145779 13926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:39745 every 8 connection(s)
I20260811 11:50:38.146689 13736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:50:38.150484 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 13736
I20260811 11:50:38.150638 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:50:38.150964 3957 external_mini_cluster.cc:1428] Setting key cb4ecc86e0ba73a572ec5cbeb21c8a0a
Aug 11 11:50:38 dist-test-slave-bndk krb5kdc[13156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449038, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:38.160759 13287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:38.149402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:44879 (local address 127.3.221.126:32993)
0811 11:50:38.149528 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:38.149532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:38.149664 (+ 132us) server_negotiation.cc:407] Connection header received
0811 11:50:38.151425 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:38.151429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:38.151491 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:38.151579 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:38.152922 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:38.153509 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:38.154656 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:38.154881 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:38.154969 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:38.157470 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:38.157473 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:38.157475 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:38.157501 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:38.158690 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:38.159202 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:38.159205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:38.159207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:38.159294 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:38.159625 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:38.159629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:38.159630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:38.159842 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:38.159935 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:38.160462 (+ 527us) server_negotiation.cc:299] Negotiation successful
0811 11:50:38.160593 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 11:50:38.161638 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:38.148940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:32993 (local address 127.3.221.67:44879)
0811 11:50:38.149506 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:38.149563 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:38.151164 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:38.151789 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:38.151797 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:38.152256 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:38.152774 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:38.152791 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:38.153636 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:38.153639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:38.154535 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:38.154545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:38.155036 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:38.155041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:38.155198 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:38.155760 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:38.155780 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:38.157302 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:38.158808 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:38.158814 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:38.158832 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:38.159102 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:38.159383 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:38.159386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:38.159388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:38.159534 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:38.159951 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:38.159959 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:38.160214 (+ 255us) client_negotiation.cc:769] Sending connection context
0811 11:50:38.160399 (+ 185us) client_negotiation.cc:240] Negotiation successful
0811 11:50:38.160732 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":129,"threads_started":1}
I20260811 11:50:38.162796 13927 heartbeater.cc:347] Connected to a master server at 127.3.221.126:32993
I20260811 11:50:38.163028 13927 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:38.163516 13927 heartbeater.cc:511] Master 127.3.221.126:32993 requested a full tablet report, sending...
I20260811 11:50:38.164637 13222 ts_manager.cc:194] Registered new tserver with Master: 8f38279f5ff048309df43e7568a5b890 (127.3.221.67:39745)
I20260811 11:50:38.165321 13222 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:44879
I20260811 11:50:38.176883 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/test-admin.keytab.
Aug 11 11:50:38 dist-test-slave-bndk krb5kdc[13156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:38.194682 3957 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260811 11:50:38.197433 13939 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:38.196076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.3.221.65:43459)
0811 11:50:38.196363 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:38.196370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:38.196396 (+ 26us) server_negotiation.cc:407] Connection header received
0811 11:50:38.196448 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:38.196453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:38.196642 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:38.196776 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:38.197251 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41982: BlockingRecv error: recv got EOF from 127.0.0.1:41982 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
W20260811 11:50:38.197667 3957 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/krb5cc)
Aug 11 11:50:38 dist-test-slave-bndk krb5kdc[13156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:38.198653 3957 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 11:50:38 dist-test-slave-bndk krb5kdc[13156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:38.199558 3957 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 11:50:38 dist-test-slave-bndk krb5kdc[13156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.65@KRBTEST.COM
I20260811 11:50:38.209129 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:38.200528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41994 (local address 127.3.221.65:43459)
0811 11:50:38.200654 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:38.200659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:38.200674 (+ 15us) server_negotiation.cc:407] Connection header received
0811 11:50:38.200807 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:38.200811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:38.200864 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:38.200932 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:38.201505 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:38.202255 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:38.203082 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:38.203476 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:38.203573 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:38.205801 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:38.205814 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:38.205829 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:38.205863 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:38.207265 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:38.207691 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:38.207697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:38.207698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:38.207783 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:38.208082 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:38.208085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:38.208087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:38.208358 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:38.208583 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:38.208813 (+ 230us) server_negotiation.cc:299] Negotiation successful
0811 11:50:38.208975 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260811 11:50:38.213033 13939 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:38.212065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41998 (local address 127.3.221.65:43459)
0811 11:50:38.212188 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:38.212192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:38.212263 (+ 71us) server_negotiation.cc:407] Connection header received
0811 11:50:38.212408 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:38.212411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:38.212453 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:38.212516 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:38.212936 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41998: BlockingRecv error: recv got EOF from 127.0.0.1:41998 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260811 11:50:38.213196 3957 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/krb5cc' of type FILE
Aug 11 11:50:38 dist-test-slave-bndk krb5kdc[13156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:38.214403 3957 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 11:50:38 dist-test-slave-bndk krb5kdc[13156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:38.215359 3957 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 11:50:38 dist-test-slave-bndk krb5kdc[13156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.65@KRBTEST.COM
I20260811 11:50:38.223680 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:38.216173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42014 (local address 127.3.221.65:43459)
0811 11:50:38.216279 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:38.216282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:38.216420 (+ 138us) server_negotiation.cc:407] Connection header received
0811 11:50:38.216552 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:38.216555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:38.216598 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:38.216670 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:38.217302 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:38.217848 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:38.218631 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:38.218835 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:38.218913 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:38.221088 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:38.221091 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:38.221093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:38.221116 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:38.222312 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:38.222697 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:38.222700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:38.222701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:38.222774 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:38.223015 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:38.223017 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:38.223018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:38.223169 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:38.223283 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:38.223427 (+ 144us) server_negotiation.cc:299] Negotiation successful
0811 11:50:38.223523 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260811 11:50:38.226320 13939 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:38.225346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42026 (local address 127.3.221.65:43459)
0811 11:50:38.225455 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:38.225459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:38.225610 (+ 151us) server_negotiation.cc:407] Connection header received
0811 11:50:38.225729 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:38.225731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:38.225772 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:38.225831 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:38.226211 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42026: BlockingRecv error: recv got EOF from 127.0.0.1:42026 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260811 11:50:38.226438 3957 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786449004114841-3957-0/krb5kdc/krb5cc' of type FILE
Aug 11 11:50:38 dist-test-slave-bndk krb5kdc[13156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:38.227398 3957 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 11:50:38 dist-test-slave-bndk krb5kdc[13156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:38.228204 3957 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 11:50:38 dist-test-slave-bndk krb5kdc[13156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.65@KRBTEST.COM
I20260811 11:50:38.235570 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:38.228967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42032 (local address 127.3.221.65:43459)
0811 11:50:38.229079 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:38.229082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:38.229160 (+ 78us) server_negotiation.cc:407] Connection header received
0811 11:50:38.229290 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:38.229293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:38.229334 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:38.229396 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:38.229819 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:38.230298 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:38.231000 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:38.231200 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:38.231283 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:38.233257 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:38.233261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:38.233263 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:38.233286 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:38.234354 (+ 1068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:38.234714 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:38.234717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:38.234718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:38.234782 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:38.234959 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:38.234961 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:38.234963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:38.235107 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:38.235241 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:38.235363 (+ 122us) server_negotiation.cc:299] Negotiation successful
0811 11:50:38.235454 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 11:50:38.236553 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 13319
I20260811 11:50:38.260797 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 13532
I20260811 11:50:38.270880 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 13736
I20260811 11:50:38.284724 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 13173
2026-08-11T11:50:38Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2430 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:50:38 dist-test-slave-bndk krb5kdc[13968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:50:38 dist-test-slave-bndk krb5kdc[13968](info): set up 2 sockets
Aug 11 11:50:38 dist-test-slave-bndk krb5kdc[13968](info): commencing operation
krb5kdc: starting...
W20260811 11:50:39.453868 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:50:39 dist-test-slave-bndk krb5kdc[13968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:50:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:50:39Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.126.keytab.
Entry for principal oryx/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.126.keytab.
I20260811 11:50:39.584293 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:46443
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.126.keytab
--principal=oryx/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:38277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:46443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:39.703073 13985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:39.703377 13985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:39.703439 13985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:39.707580 13985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:50:39.707655 13985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:39.707681 13985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:39.707703 13985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:50:39.707732 13985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:50:39.713413 13985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:38277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:46443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.126.keytab
--principal=oryx/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:46443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:39.714519 13985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:39.715569 13985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:50:39.724673 13985 server_base.cc:1034] running on GCE node
W20260811 11:50:39.724627 13996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:39.724635 13997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:39.724627 14001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:39.725584 13985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:39.726691 13985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:39.727852 13985 hybrid_clock.cc:648] HybridClock initialized: now 1786449039727836 us; error 30 us; skew 500 ppm
Aug 11 11:50:39 dist-test-slave-bndk krb5kdc[13968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449039, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:39.730710 13985 init.cc:377] Logged in from keytab as oryx/127.3.221.126@KRBTEST.COM (short username oryx)
I20260811 11:50:39.731724 13985 webserver.cc:460] Webserver started at http://127.3.221.126:35585/ using document root <none> and password file <none>
I20260811 11:50:39.732290 13985 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:39.732354 13985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:39.732539 13985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:39.734791 13985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "9b2b536c69864468947c78d7f24fea04"
format_stamp: "Formatted at 2026-08-11 11:50:39 on dist-test-slave-bndk"
server_key: "1ae829c3c079fc5b131bcd77f7730a73"
server_key_iv: "941bb19ae92332e13696d8555ca541f8"
server_key_version: "encryptionkey@0"
I20260811 11:50:39.735356 13985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "9b2b536c69864468947c78d7f24fea04"
format_stamp: "Formatted at 2026-08-11 11:50:39 on dist-test-slave-bndk"
server_key: "1ae829c3c079fc5b131bcd77f7730a73"
server_key_iv: "941bb19ae92332e13696d8555ca541f8"
server_key_version: "encryptionkey@0"
I20260811 11:50:39.739280 13985 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260811 11:50:39.741796 14010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:39.743077 13985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 11:50:39.743213 13985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "9b2b536c69864468947c78d7f24fea04"
format_stamp: "Formatted at 2026-08-11 11:50:39 on dist-test-slave-bndk"
server_key: "1ae829c3c079fc5b131bcd77f7730a73"
server_key_iv: "941bb19ae92332e13696d8555ca541f8"
server_key_version: "encryptionkey@0"
I20260811 11:50:39.743319 13985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:39.758899 13985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:39.761991 13985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:39.762190 13985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:39.769451 13985 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:46443
I20260811 11:50:39.769714 14086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:46443 every 8 connection(s)
I20260811 11:50:39.770553 13985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:50:39.772055 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 13985
I20260811 11:50:39.772186 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:50:39.772461 3957 external_mini_cluster.cc:1428] Setting key 30c203e9ea53d6713931e75ddd592059
I20260811 11:50:39.773910 14088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:39.779903 14088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04: Bootstrap starting.
Aug 11 11:50:39 dist-test-slave-bndk krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM
I20260811 11:50:39.782371 14088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:39.783200 14088 log.cc:824] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:39.785153 14088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04: No bootstrap required, opened a new log
I20260811 11:50:39.786166 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:39.773918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49926 (local address 127.3.221.126:46443)
0811 11:50:39.774414 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:39.774431 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:39.774470 (+ 39us) server_negotiation.cc:407] Connection header received
0811 11:50:39.775147 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:39.775164 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:39.775509 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:39.775827 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:39.776664 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:39.777729 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:39.778554 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:39.779022 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:39.779165 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:39.781714 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:39.781720 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:39.781733 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:39.781776 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:39.783404 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:39.783836 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:39.783841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:39.783845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:39.783957 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:39.784222 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:39.784224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:39.784225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:39.784627 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:39.784817 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:39.785100 (+ 283us) server_negotiation.cc:299] Negotiation successful
0811 11:50:39.785437 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":148,"threads_started":1}
I20260811 11:50:39.789206 14088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } }
I20260811 11:50:39.789494 14088 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:39.789566 14088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b2b536c69864468947c78d7f24fea04, State: Initialized, Role: FOLLOWER
I20260811 11:50:39.789990 14088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } }
I20260811 11:50:39.790125 14088 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:50:39.790191 14088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:50:39.790318 14088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:39.791292 14088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } }
I20260811 11:50:39.791638 14088 leader_election.cc:302] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b2b536c69864468947c78d7f24fea04; no voters:
I20260811 11:50:39.792001 14088 leader_election.cc:288] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:50:39.792152 14096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:39.792380 14096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [term 1 LEADER]: Becoming Leader. State: Replica: 9b2b536c69864468947c78d7f24fea04, State: Running, Role: LEADER
I20260811 11:50:39.792691 14096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } }
I20260811 11:50:39.793301 14088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:50:39.794498 14098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b2b536c69864468947c78d7f24fea04. Latest consensus state: current_term: 1 leader_uuid: "9b2b536c69864468947c78d7f24fea04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } }
I20260811 11:50:39.794533 14097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b2b536c69864468947c78d7f24fea04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } }
I20260811 11:50:39.794634 14098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:39.794661 14097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:39.795012 14111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:50:39.798434 14111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:50:39.805100 14111 catalog_manager.cc:1347] Generated new cluster ID: 93c1574656b04143adc85e20ebe4d8ad
I20260811 11:50:39.805189 14111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:50:39.829312 14111 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:50:39.830662 14111 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:50:39.835988 14111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04: Generated new TSK 0
I20260811 11:50:39.836674 14111 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.65.keytab.
Entry for principal oryx/127.3.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.65.keytab.
I20260811 11:50:39.869724 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:46443
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.65.keytab
--principal=oryx/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:38277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:39.988758 14126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:39.989039 14126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:39.989100 14126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:39.993170 14126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:39.993278 14126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:39.993373 14126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:50:39.998960 14126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:38277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.65.keytab
--principal=oryx/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:46443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:40.000139 14126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:40.001199 14126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:40.011127 14144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:40.011127 14138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:40.011158 14135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:40.011158 14126 server_base.cc:1034] running on GCE node
I20260811 11:50:40.011765 14126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:40.012392 14126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:40.013549 14126 hybrid_clock.cc:648] HybridClock initialized: now 1786449040013527 us; error 31 us; skew 500 ppm
Aug 11 11:50:40 dist-test-slave-bndk krb5kdc[13968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449040, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:40.016413 14126 init.cc:377] Logged in from keytab as oryx/127.3.221.65@KRBTEST.COM (short username oryx)
I20260811 11:50:40.017365 14126 webserver.cc:460] Webserver started at http://127.3.221.65:41587/ using document root <none> and password file <none>
I20260811 11:50:40.017930 14126 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:40.017999 14126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:40.018206 14126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:40.020287 14126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "6065b7065aa84491a806b7276eb7b333"
format_stamp: "Formatted at 2026-08-11 11:50:40 on dist-test-slave-bndk"
server_key: "99c97e09a7706844e2c3032d92583c84"
server_key_iv: "a1f7b6b6bedd5dd4e5d1183b62f320be"
server_key_version: "encryptionkey@0"
I20260811 11:50:40.020843 14126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "6065b7065aa84491a806b7276eb7b333"
format_stamp: "Formatted at 2026-08-11 11:50:40 on dist-test-slave-bndk"
server_key: "99c97e09a7706844e2c3032d92583c84"
server_key_iv: "a1f7b6b6bedd5dd4e5d1183b62f320be"
server_key_version: "encryptionkey@0"
I20260811 11:50:40.024627 14126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260811 11:50:40.027225 14153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:40.028548 14126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 11:50:40.028666 14126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "6065b7065aa84491a806b7276eb7b333"
format_stamp: "Formatted at 2026-08-11 11:50:40 on dist-test-slave-bndk"
server_key: "99c97e09a7706844e2c3032d92583c84"
server_key_iv: "a1f7b6b6bedd5dd4e5d1183b62f320be"
server_key_version: "encryptionkey@0"
I20260811 11:50:40.028774 14126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:40.038441 14126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:40.041488 14126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:40.041680 14126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:40.042299 14126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:40.043177 14126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:40.043234 14126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:40.043292 14126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:40.043318 14126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:40.052196 14126 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:45063
I20260811 11:50:40.052218 14324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:45063 every 8 connection(s)
I20260811 11:50:40.053125 14126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:50:40.057688 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 14126
I20260811 11:50:40.057806 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:50:40.058074 3957 external_mini_cluster.cc:1428] Setting key b3e354238d5a426ec8e92907b87216ae
Aug 11 11:50:40 dist-test-slave-bndk krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449040, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM
I20260811 11:50:40.067473 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:40.054993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:56723 (local address 127.3.221.126:46443)
0811 11:50:40.055164 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:40.055169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:40.055988 (+ 819us) server_negotiation.cc:407] Connection header received
0811 11:50:40.057264 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:40.057268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:40.057329 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:40.057406 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:40.058755 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.059359 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.060595 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.060909 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.061011 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:40.063649 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:40.063652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:40.063655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:40.063683 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:40.065307 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:40.065822 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:40.065827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:40.065829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:40.065933 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:40.066242 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:40.066246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:40.066248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:40.066516 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:40.066631 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:40.067130 (+ 499us) server_negotiation.cc:299] Negotiation successful
0811 11:50:40.067282 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 11:50:40.068249 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:40.055382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.65:56723)
0811 11:50:40.055829 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:40.055885 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:40.057014 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:40.057570 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:40.057577 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:40.058012 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:40.058580 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:40.058600 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.059503 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.059506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:40.060460 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:40.060471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.061051 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.061054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:40.061193 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:40.061825 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:40.061842 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:40.063473 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:40.065427 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:40.065433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:40.065448 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:40.065716 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:40.066019 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:40.066021 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:40.066023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:40.066156 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:40.066613 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:40.066617 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:40.066886 (+ 269us) client_negotiation.cc:769] Sending connection context
0811 11:50:40.067050 (+ 164us) client_negotiation.cc:240] Negotiation successful
0811 11:50:40.067328 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for oryx/127.3.221.66@KRBTEST.COM; defaulting to no policy
I20260811 11:50:40.069593 14325 heartbeater.cc:347] Connected to a master server at 127.3.221.126:46443
I20260811 11:50:40.069832 14325 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:40.070358 14325 heartbeater.cc:511] Master 127.3.221.126:46443 requested a full tablet report, sending...
I20260811 11:50:40.072065 14034 ts_manager.cc:194] Registered new tserver with Master: 6065b7065aa84491a806b7276eb7b333 (127.3.221.65:45063)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.221.66@KRBTEST.COM" created.
I20260811 11:50:40.073329 14034 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:56723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.66.keytab.
Entry for principal oryx/127.3.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.66.keytab.
I20260811 11:50:40.094738 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:46443
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.66.keytab
--principal=oryx/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:38277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:40.210409 14332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:40.210690 14332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:40.210736 14332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:40.214828 14332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:40.214905 14332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:40.214972 14332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:50:40.220494 14332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:38277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.66.keytab
--principal=oryx/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:46443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:40.221681 14332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:40.222738 14332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:40.232743 14340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:40.232748 14341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:40.232748 14345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:40.233173 14332 server_base.cc:1034] running on GCE node
I20260811 11:50:40.233606 14332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:40.234253 14332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:40.235421 14332 hybrid_clock.cc:648] HybridClock initialized: now 1786449040235399 us; error 35 us; skew 500 ppm
Aug 11 11:50:40 dist-test-slave-bndk krb5kdc[13968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449040, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:40.238188 14332 init.cc:377] Logged in from keytab as oryx/127.3.221.66@KRBTEST.COM (short username oryx)
I20260811 11:50:40.239089 14332 webserver.cc:460] Webserver started at http://127.3.221.66:39259/ using document root <none> and password file <none>
I20260811 11:50:40.239645 14332 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:40.239717 14332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:40.239938 14332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:40.242172 14332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "09017061acfe48f6bb57b06e16ce084e"
format_stamp: "Formatted at 2026-08-11 11:50:40 on dist-test-slave-bndk"
server_key: "2281dd4f670e08847f6fdcbff413ea07"
server_key_iv: "ffb82508047ddba81259adc3714279db"
server_key_version: "encryptionkey@0"
I20260811 11:50:40.242769 14332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "09017061acfe48f6bb57b06e16ce084e"
format_stamp: "Formatted at 2026-08-11 11:50:40 on dist-test-slave-bndk"
server_key: "2281dd4f670e08847f6fdcbff413ea07"
server_key_iv: "ffb82508047ddba81259adc3714279db"
server_key_version: "encryptionkey@0"
I20260811 11:50:40.246654 14332 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:50:40.249321 14358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:40.250658 14332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 11:50:40.250768 14332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "09017061acfe48f6bb57b06e16ce084e"
format_stamp: "Formatted at 2026-08-11 11:50:40 on dist-test-slave-bndk"
server_key: "2281dd4f670e08847f6fdcbff413ea07"
server_key_iv: "ffb82508047ddba81259adc3714279db"
server_key_version: "encryptionkey@0"
I20260811 11:50:40.250890 14332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:40.269050 14332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:40.272495 14332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:40.272691 14332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:40.273348 14332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:40.274223 14332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:40.274282 14332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:40.274338 14332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:40.274362 14332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:40.282784 14332 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:40795
I20260811 11:50:40.282810 14536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:40795 every 8 connection(s)
I20260811 11:50:40.283695 14332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:50:40.292805 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 14332
I20260811 11:50:40.292914 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
Aug 11 11:50:40 dist-test-slave-bndk krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449040, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM
I20260811 11:50:40.293207 3957 external_mini_cluster.cc:1428] Setting key 08abf7654d2422ae5545f695de39c02d
I20260811 11:50:40.298038 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:40.285581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:49859 (local address 127.3.221.126:46443)
0811 11:50:40.285753 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:40.285759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:40.286541 (+ 782us) server_negotiation.cc:407] Connection header received
0811 11:50:40.287813 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:40.287816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:40.287876 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:40.287951 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:40.289190 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.289760 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.290886 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.291114 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.291213 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:40.293909 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:40.293913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:40.293916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:40.293947 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:40.295809 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:40.296336 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:40.296339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:40.296347 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:40.296437 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:40.296795 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:40.296799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:40.296800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:40.297047 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:40.297133 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:40.297790 (+ 657us) server_negotiation.cc:299] Negotiation successful
0811 11:50:40.297915 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 11:50:40.299068 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:40.285916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.66:49859)
0811 11:50:40.286378 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:40.286423 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:40.287597 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:40.288136 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:40.288144 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:40.288526 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:40.289042 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:40.289057 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.289884 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.289887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:40.290766 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:40.290776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.291275 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.291279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:40.291457 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:40.292028 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:40.292047 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:40.293749 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:40.295934 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:40.295940 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:40.295955 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:40.296232 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:40.296527 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:40.296530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:40.296532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:40.296678 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:40.297147 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:40.297154 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:40.297526 (+ 372us) client_negotiation.cc:769] Sending connection context
0811 11:50:40.297748 (+ 222us) client_negotiation.cc:240] Negotiation successful
0811 11:50:40.298134 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":116,"threads_started":1}
I20260811 11:50:40.300205 14537 heartbeater.cc:347] Connected to a master server at 127.3.221.126:46443
I20260811 11:50:40.300495 14537 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:40.301128 14537 heartbeater.cc:511] Master 127.3.221.126:46443 requested a full tablet report, sending...
I20260811 11:50:40.302263 14034 ts_manager.cc:194] Registered new tserver with Master: 09017061acfe48f6bb57b06e16ce084e (127.3.221.66:40795)
I20260811 11:50:40.302842 14034 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:49859
WARNING: no policy specified for oryx/127.3.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.67.keytab.
Entry for principal oryx/127.3.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.67.keytab.
I20260811 11:50:40.324375 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:46443
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.67.keytab
--principal=oryx/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:38277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:40.440143 14544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:40.440436 14544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:40.440479 14544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:40.444458 14544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:40.444523 14544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:40.444582 14544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:50:40.449888 14544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:38277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.67.keytab
--principal=oryx/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:46443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:40.450934 14544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:40.451957 14544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:50:40.461884 14544 server_base.cc:1034] running on GCE node
W20260811 11:50:40.461807 14550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:40.461798 14551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:40.461810 14554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:40.462551 14544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:40.463212 14544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:40.464417 14544 hybrid_clock.cc:648] HybridClock initialized: now 1786449040464384 us; error 43 us; skew 500 ppm
Aug 11 11:50:40 dist-test-slave-bndk krb5kdc[13968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449040, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:40.467242 14544 init.cc:377] Logged in from keytab as oryx/127.3.221.67@KRBTEST.COM (short username oryx)
I20260811 11:50:40.468217 14544 webserver.cc:460] Webserver started at http://127.3.221.67:41303/ using document root <none> and password file <none>
I20260811 11:50:40.468794 14544 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:40.468864 14544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:40.469071 14544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:40.471345 14544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "777856fcd5864cffa63bd364422fc3a8"
format_stamp: "Formatted at 2026-08-11 11:50:40 on dist-test-slave-bndk"
server_key: "ff5157be589d6aaa04bcea18f2bc6033"
server_key_iv: "49ab9b01e192043427145a864320d5d9"
server_key_version: "encryptionkey@0"
I20260811 11:50:40.471940 14544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "777856fcd5864cffa63bd364422fc3a8"
format_stamp: "Formatted at 2026-08-11 11:50:40 on dist-test-slave-bndk"
server_key: "ff5157be589d6aaa04bcea18f2bc6033"
server_key_iv: "49ab9b01e192043427145a864320d5d9"
server_key_version: "encryptionkey@0"
I20260811 11:50:40.475889 14544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260811 11:50:40.478549 14568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:40.479799 14544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 11:50:40.479921 14544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "777856fcd5864cffa63bd364422fc3a8"
format_stamp: "Formatted at 2026-08-11 11:50:40 on dist-test-slave-bndk"
server_key: "ff5157be589d6aaa04bcea18f2bc6033"
server_key_iv: "49ab9b01e192043427145a864320d5d9"
server_key_version: "encryptionkey@0"
I20260811 11:50:40.480026 14544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:40.493937 14544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:40.497424 14544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:40.497620 14544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:40.498227 14544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:40.499292 14544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:40.499346 14544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:40.499390 14544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:40.499419 14544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:40.507977 14544 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:44329
I20260811 11:50:40.508028 14768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:44329 every 8 connection(s)
I20260811 11:50:40.508925 14544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:50:40.512006 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 14544
I20260811 11:50:40.512142 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:50:40.512432 3957 external_mini_cluster.cc:1428] Setting key d57b7d9472b740802e96c032d8964a19
Aug 11 11:50:40 dist-test-slave-bndk krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449040, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM
I20260811 11:50:40.523213 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:40.510802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:33129 (local address 127.3.221.126:46443)
0811 11:50:40.510989 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:40.510993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:40.511909 (+ 916us) server_negotiation.cc:407] Connection header received
0811 11:50:40.513261 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:40.513264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:40.513327 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:40.513413 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:40.514879 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.515444 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.516987 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.517323 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.517410 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:40.519851 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:40.519854 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:40.519856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:40.519886 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:40.521275 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:40.521901 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:40.521911 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:40.521913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:40.521986 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:40.522262 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:40.522264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:40.522265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:40.522440 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:40.522506 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:40.522977 (+ 471us) server_negotiation.cc:299] Negotiation successful
0811 11:50:40.523084 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 11:50:40.524091 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:40.511252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.67:33129)
0811 11:50:40.511754 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:40.511799 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:40.513009 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:40.513570 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:40.513580 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:40.514077 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:40.514740 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:40.514756 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.515546 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.515548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:40.516836 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:40.516850 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.517471 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.517474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:40.517634 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:40.518171 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:40.518187 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:40.519665 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:40.521407 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:40.521413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:40.521427 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:40.521806 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:40.522055 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:40.522058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:40.522059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:40.522199 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:40.522517 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:40.522522 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:40.522768 (+ 246us) client_negotiation.cc:769] Sending connection context
0811 11:50:40.522940 (+ 172us) client_negotiation.cc:240] Negotiation successful
0811 11:50:40.523212 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":130,"threads_started":1}
I20260811 11:50:40.525377 14769 heartbeater.cc:347] Connected to a master server at 127.3.221.126:46443
I20260811 11:50:40.525648 14769 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:40.526217 14769 heartbeater.cc:511] Master 127.3.221.126:46443 requested a full tablet report, sending...
I20260811 11:50:40.527313 14034 ts_manager.cc:194] Registered new tserver with Master: 777856fcd5864cffa63bd364422fc3a8 (127.3.221.67:44329)
I20260811 11:50:40.527897 14034 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:33129
I20260811 11:50:40.538380 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.221.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.221.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.221.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.125.keytab.
Entry for principal oryx/127.3.221.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.125.keytab.
I20260811 11:50:40.567919 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.125:39327
--webserver_interface=127.3.221.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.125.keytab
--principal=oryx/127.3.221.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:38277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:46443,127.3.221.125:39327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:40.684600 14775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:40.684885 14775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:40.684955 14775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:40.688961 14775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:50:40.689035 14775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:40.689061 14775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:40.689082 14775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:50:40.689103 14775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:50:40.694730 14775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:38277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:46443,127.3.221.125:39327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.125.keytab
--principal=oryx/127.3.221.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.125:39327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.221.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
W20260811 11:50:40.694990 14775 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.221.126:46443,127.3.221.125:39327'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260811 11:50:40.695947 14775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:40.696985 14775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:40.705529 14784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:40.705555 14785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:40.705731 14775 server_base.cc:1034] running on GCE node
W20260811 11:50:40.705538 14788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:40.706451 14775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:40.707556 14775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:40.708714 14775 hybrid_clock.cc:648] HybridClock initialized: now 1786449040708696 us; error 28 us; skew 500 ppm
Aug 11 11:50:40 dist-test-slave-bndk krb5kdc[13968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449040, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:40.711469 14775 init.cc:377] Logged in from keytab as oryx/127.3.221.125@KRBTEST.COM (short username oryx)
I20260811 11:50:40.712402 14775 webserver.cc:460] Webserver started at http://127.3.221.125:40341/ using document root <none> and password file <none>
I20260811 11:50:40.712962 14775 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:40.713032 14775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:40.713274 14775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:40.715405 14775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/data/instance:
uuid: "30ce91da8cce49dba59fb1865beb961c"
format_stamp: "Formatted at 2026-08-11 11:50:40 on dist-test-slave-bndk"
server_key: "7fb6a27c676d25b0eac064e4609d57c1"
server_key_iv: "3a6e6b90c41dc91ccfb25b18b670652b"
server_key_version: "encryptionkey@0"
I20260811 11:50:40.715968 14775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/wal/instance:
uuid: "30ce91da8cce49dba59fb1865beb961c"
format_stamp: "Formatted at 2026-08-11 11:50:40 on dist-test-slave-bndk"
server_key: "7fb6a27c676d25b0eac064e4609d57c1"
server_key_iv: "3a6e6b90c41dc91ccfb25b18b670652b"
server_key_version: "encryptionkey@0"
I20260811 11:50:40.719782 14775 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260811 11:50:40.722436 14798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:40.723712 14775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 11:50:40.723858 14775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/wal
uuid: "30ce91da8cce49dba59fb1865beb961c"
format_stamp: "Formatted at 2026-08-11 11:50:40 on dist-test-slave-bndk"
server_key: "7fb6a27c676d25b0eac064e4609d57c1"
server_key_iv: "3a6e6b90c41dc91ccfb25b18b670652b"
server_key_version: "encryptionkey@0"
I20260811 11:50:40.723966 14775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:40.742359 14775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:40.745132 14775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:40.745347 14775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:40.752313 14775 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.125:39327
I20260811 11:50:40.752595 14871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.125:39327 every 8 connection(s)
I20260811 11:50:40.753378 14775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/data/info.pb
I20260811 11:50:40.755666 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 14775
I20260811 11:50:40.756522 14872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:40.760732 14872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:50:40 dist-test-slave-bndk krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.221.125@KRBTEST.COM
Aug 11 11:50:40 dist-test-slave-bndk krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449040, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.125@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM
I20260811 11:50:40.771332 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:40.757421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47544 (local address 127.3.221.125:39327)
0811 11:50:40.757881 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:40.757897 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:40.757933 (+ 36us) server_negotiation.cc:407] Connection header received
0811 11:50:40.758598 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:40.758615 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:40.758944 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:40.759274 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:40.760118 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.761121 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.762324 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.763010 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.763165 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:40.765979 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:40.765987 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:40.766000 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:40.766035 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:40.767765 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:40.768199 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:40.768205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:40.768212 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:40.768353 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:40.768705 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:40.768707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:40.768708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:40.769209 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:40.769510 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:40.769994 (+ 484us) server_negotiation.cc:299] Negotiation successful
0811 11:50:40.770305 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":148,"threads_started":1}
I20260811 11:50:40.772329 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:40.762743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49942 (local address 127.3.221.126:46443)
0811 11:50:40.762857 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:40.762860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:40.763238 (+ 378us) server_negotiation.cc:407] Connection header received
0811 11:50:40.763498 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:40.763502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:40.763551 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:40.763616 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:40.764342 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.764994 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.766023 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.766362 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.766444 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:40.768622 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:40.768632 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:40.768635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:40.768667 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:40.770248 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:40.770723 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:40.770727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:40.770729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:40.770822 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:40.771122 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:40.771125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:40.771127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:40.771408 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:40.771506 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:40.772060 (+ 554us) server_negotiation.cc:299] Negotiation successful
0811 11:50:40.772186 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 11:50:40.772852 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:40.762699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.0.0.1:49942)
0811 11:50:40.763102 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:40.763155 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:40.763336 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:40.763702 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:40.763708 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:40.764066 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:40.764198 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:40.764204 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.765145 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.765148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:40.765902 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:40.765922 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.766466 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.766470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:40.766593 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:40.767028 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:40.767045 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:40.768468 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:40.770344 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:40.770351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:40.770357 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:40.770618 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:40.770929 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:40.770932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:40.770934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:40.771029 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:40.771563 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:40.771569 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:40.771823 (+ 254us) client_negotiation.cc:769] Sending connection context
0811 11:50:40.772475 (+ 652us) client_negotiation.cc:240] Negotiation successful
0811 11:50:40.772668 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":167,"threads_started":1}
I20260811 11:50:40.774549 14872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:50:40 dist-test-slave-bndk krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449040, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.125@KRBTEST.COM for oryx/127.3.221.125@KRBTEST.COM
I20260811 11:50:40.782744 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:40.775014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:39327 (local address 127.0.0.1:47550)
0811 11:50:40.775113 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:40.775125 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:40.775196 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:40.775376 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:40.775378 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:40.775542 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:40.775658 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:40.775664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.776716 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.776719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:40.777553 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:40.777564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.777970 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.777974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:40.778061 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:40.778488 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:40.778501 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:40.779825 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:40.781390 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:40.781394 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:40.781396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:40.781626 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:40.781843 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:40.781846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:40.781848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:40.781926 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:40.782227 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:40.782230 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:40.782323 (+ 93us) client_negotiation.cc:769] Sending connection context
0811 11:50:40.782423 (+ 100us) client_negotiation.cc:240] Negotiation successful
0811 11:50:40.782559 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260811 11:50:40.782749 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:40.775114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47550 (local address 127.3.221.125:39327)
0811 11:50:40.775216 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:40.775219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:40.775230 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:50:40.775258 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:40.775260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:40.775293 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:40.775354 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:40.775755 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.776617 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.777656 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.777868 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.777945 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:40.779979 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:40.779982 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:40.779984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:40.780007 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:40.781285 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:40.781705 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:40.781707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:40.781708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:40.781773 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:40.781995 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:40.781997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:40.781998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:40.782159 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:40.782226 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:40.782469 (+ 243us) server_negotiation.cc:299] Negotiation successful
0811 11:50:40.782617 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":37}
I20260811 11:50:40.785924 14872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c: Bootstrap starting.
I20260811 11:50:40.788300 14872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:40.789034 14872 log.cc:824] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:40.791031 14872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c: No bootstrap required, opened a new log
I20260811 11:50:40.794296 14872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } }
I20260811 11:50:40.794553 14872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:40.794627 14872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ce91da8cce49dba59fb1865beb961c, State: Initialized, Role: FOLLOWER
I20260811 11:50:40.795092 14872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } }
I20260811 11:50:40.795610 14883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } } }
I20260811 11:50:40.795729 14883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:50:40.796377 14872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [sys.catalog]: configured and running, proceeding with master startup.
W20260811 11:50:40.801169 14902 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:50:40.801298 14902 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:50:40.803102 14775 master_runner.cc:428] Detected that this master 30ce91da8cce49dba59fb1865beb961c is joining an existing cluster
I20260811 11:50:40.803159 14775 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.221.125:39327
I20260811 11:50:40.813526 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:40.805276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47564 (local address 127.3.221.125:39327)
0811 11:50:40.805371 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:40.805374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:40.805983 (+ 609us) server_negotiation.cc:407] Connection header received
0811 11:50:40.806110 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:40.806113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:40.806150 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:40.806193 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:40.806832 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.807591 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.808834 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.809158 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.809291 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:40.810487 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:40.810491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:40.810494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:40.810515 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:40.811881 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:40.812262 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:40.812266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:40.812267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:40.812426 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:40.812687 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:40.812690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:40.812692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:40.812863 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:40.812955 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:40.813186 (+ 231us) server_negotiation.cc:299] Negotiation successful
0811 11:50:40.813363 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":27,"spinlock_wait_cycles":1792}
I20260811 11:50:40.813541 14917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:40.805169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:39327 (local address 127.0.0.1:47564)
0811 11:50:40.805897 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:40.805918 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:40.806018 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:40.806320 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:40.806323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:40.806582 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:40.806727 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:40.806734 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.807716 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.807720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:40.808710 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:40.808721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.809295 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.809299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:40.809410 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:40.809797 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:40.809813 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:40.810353 (+ 540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:40.811987 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:40.811990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:40.811993 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:40.812175 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:40.812514 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:40.812517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:40.812519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:40.812607 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:40.812947 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:40.812949 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:40.813045 (+ 96us) client_negotiation.cc:769] Sending connection context
0811 11:50:40.813131 (+ 86us) client_negotiation.cc:240] Negotiation successful
0811 11:50:40.813315 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":47,"thread_start_us":119,"threads_started":1}
I20260811 11:50:40.813917 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:40.805706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49954 (local address 127.3.221.126:46443)
0811 11:50:40.805817 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:40.805822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:40.806080 (+ 258us) server_negotiation.cc:407] Connection header received
0811 11:50:40.806195 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:40.806198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:40.806245 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:40.806318 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:40.806950 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.807703 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.808553 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.808776 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.808847 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:40.810616 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:40.810619 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:40.810622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:40.810651 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:40.812150 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:40.812596 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:40.812599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:40.812601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:40.812691 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:40.812982 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:40.812986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:40.812987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:40.813197 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:40.813293 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:40.813637 (+ 344us) server_negotiation.cc:299] Negotiation successful
0811 11:50:40.813759 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260811 11:50:40.814433 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:40.805635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.0.0.1:49954)
0811 11:50:40.805986 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:40.806001 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:40.806105 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:40.806368 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:40.806372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:40.806631 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:40.806817 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:40.806825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.807794 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.807797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:40.808416 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:40.808428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.809018 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.809021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:40.809147 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:40.810073 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:40.810084 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:40.810476 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:40.812256 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:40.812259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:40.812260 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:40.812470 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:40.812792 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:40.812794 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:40.812796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:40.812883 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:40.813373 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:40.813376 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:40.813463 (+ 87us) client_negotiation.cc:769] Sending connection context
0811 11:50:40.814104 (+ 641us) client_negotiation.cc:240] Negotiation successful
0811 11:50:40.814244 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":43,"thread_start_us":104,"threads_started":1}
Aug 11 11:50:40 dist-test-slave-bndk krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449039, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.126@KRBTEST.COM for oryx/127.3.221.125@KRBTEST.COM
I20260811 11:50:40.828197 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:40.818748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47576 (local address 127.3.221.125:39327)
0811 11:50:40.818855 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:40.818858 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:40.819229 (+ 371us) server_negotiation.cc:407] Connection header received
0811 11:50:40.819424 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:40.819427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:40.819477 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:40.819543 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:40.820232 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.820741 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.821651 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.821871 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.821966 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:40.824423 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:40.824427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:40.824430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:40.824457 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:40.826082 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:40.826679 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:40.826682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:40.826684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:40.826775 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:40.827080 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:40.827084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:40.827086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:40.827399 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:40.827503 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:40.827913 (+ 410us) server_negotiation.cc:299] Negotiation successful
0811 11:50:40.828034 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260811 11:50:40.828382 14920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:40.818729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:39327 (local address 127.0.0.1:47576)
0811 11:50:40.819105 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:40.819150 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:40.819290 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:40.819597 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:40.819603 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:40.819999 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:40.820115 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:40.820121 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.820834 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.820837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:40.821538 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:40.821550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.821973 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.821977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:40.822092 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:40.822632 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:40.822645 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:40.824295 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:40.826203 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:40.826216 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:40.826224 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:40.826563 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:40.826859 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:40.826862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:40.826864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:40.826965 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:40.827521 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:40.827529 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:40.827690 (+ 161us) client_negotiation.cc:769] Sending connection context
0811 11:50:40.827846 (+ 156us) client_negotiation.cc:240] Negotiation successful
0811 11:50:40.828100 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":121,"threads_started":1}
I20260811 11:50:40.830063 14034 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.221.125:39327: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "30ce91da8cce49dba59fb1865beb961c"
member_type: NON_VOTER
last_known_addr {
host: "127.3.221.125"
port: 39327
}
attrs {
promote: true
}
}
dest_uuid: "9b2b536c69864468947c78d7f24fea04"
cas_config_opid_index: -1
I20260811 11:50:40.830796 14034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: NON_VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: true } }
I20260811 11:50:40.832484 14924 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 30ce91da8cce49dba59fb1865beb961c (127.3.221.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: NON_VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: true } } }
I20260811 11:50:40.833465 14835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:40.834589 14924 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.221.125:39327
I20260811 11:50:40.834897 14835 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 1 FOLLOWER]: Refusing update from remote peer 9b2b536c69864468947c78d7f24fea04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260811 11:50:40.835150 14922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9b2b536c69864468947c78d7f24fea04" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: NON_VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: true } } }
I20260811 11:50:40.835354 14922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:40.835695 14775 master.cc:537] Master@127.3.221.125:39327 shutting down...
I20260811 11:50:40.836161 14922 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: NON_VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 11:50:40.838605 14011 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
WARNING: no policy specified for oryx/127.3.221.124@KRBTEST.COM; defaulting to no policy
I20260811 11:50:40.841107 14775 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:40.841454 14775 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:40.841580 14775 tablet_replica.cc:330] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.221.124@KRBTEST.COM" created.
I20260811 11:50:40.849431 14775 master.cc:552] Master@127.3.221.125:39327 shutdown complete.
I20260811 11:50:40.849495 14775 master_runner.cc:305] Clearing existing system tablet
I20260811 11:50:40.849741 14775 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:40.849794 14775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:40.850346 14920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:40.849914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:39327 (local address 127.0.0.1:47580)
0811 11:50:40.850165 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:40.850242 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:39327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
W20260811 11:50:40.850687 14011 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 -> Peer 30ce91da8cce49dba59fb1865beb961c (127.3.221.125:39327): Couldn't send request to peer 30ce91da8cce49dba59fb1865beb961c. Status: Network error: Client connection negotiation failed: client connection to 127.3.221.125:39327: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260811 11:50:40.851091 14775 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:40.851963 14929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:40.852173 14775 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:40.852298 14775 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:40.852339 14775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/wal
uuid: "30ce91da8cce49dba59fb1865beb961c"
format_stamp: "Formatted at 2026-08-11 11:50:40 on dist-test-slave-bndk"
server_key: "7fb6a27c676d25b0eac064e4609d57c1"
server_key_iv: "3a6e6b90c41dc91ccfb25b18b670652b"
server_key_version: "encryptionkey@0"
I20260811 11:50:40.854619 14775 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:40.858040 14775 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260811 11:50:40.858150 14775 log.cc:1196] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260811 11:50:40.858415 14775 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c: Deleting consensus metadata
I20260811 11:50:40.859189 14775 master_runner.cc:315] Copying system tablet from 127.3.221.126:46443
I20260811 11:50:40.860743 14775 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c: tablet copy: Beginning tablet copy session from remote peer at address 127.3.221.126:46443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.221.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.124.keytab.
Entry for principal oryx/127.3.221.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.124.keytab.
I20260811 11:50:40.863649 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.124:45801
--webserver_interface=127.3.221.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.124.keytab
--principal=oryx/127.3.221.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:38277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:46443,127.3.221.125:39327,127.3.221.124:45801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:50:40.869022 14943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:40.861440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.0.0.1:49966)
0811 11:50:40.861770 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:40.861783 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:40.861900 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:40.862151 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:40.862153 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:40.862368 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:40.862501 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:40.862508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.863568 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.863571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:40.864350 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:40.864364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.864972 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.864977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:40.865071 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:40.865673 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:40.865686 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:40.866028 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:40.867601 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:40.867605 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:40.867607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:40.867809 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:40.868065 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:40.868069 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:40.868071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:40.868156 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:40.868525 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:40.868528 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:40.868624 (+ 96us) client_negotiation.cc:769] Sending connection context
0811 11:50:40.868720 (+ 96us) client_negotiation.cc:240] Negotiation successful
0811 11:50:40.868848 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":89,"threads_started":1}
I20260811 11:50:40.869124 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:40.861540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49966 (local address 127.3.221.126:46443)
0811 11:50:40.861677 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:40.861683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:40.861860 (+ 177us) server_negotiation.cc:407] Connection header received
0811 11:50:40.861976 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:40.861979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:40.862032 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:40.862127 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:40.862614 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.863434 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.864491 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:40.864843 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:40.864952 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:40.866175 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:40.866180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:40.866182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:40.866209 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:40.867497 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:40.867898 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:40.867902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:40.867904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:40.867982 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:40.868233 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:40.868237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:40.868239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:40.868434 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:40.868522 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:40.868822 (+ 300us) server_negotiation.cc:299] Negotiation successful
0811 11:50:40.868960 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 11:50:40.869808 14062 tablet_copy_service.cc:140] P 9b2b536c69864468947c78d7f24fea04: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 30ce91da8cce49dba59fb1865beb961c ({username='oryx', principal='oryx/127.3.221.125@KRBTEST.COM'} at 127.0.0.1:49966)
I20260811 11:50:40.870025 14062 tablet_copy_service.cc:161] P 9b2b536c69864468947c78d7f24fea04: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 30ce91da8cce49dba59fb1865beb961c at {username='oryx', principal='oryx/127.3.221.125@KRBTEST.COM'} at 127.0.0.1:49966: session id = 30ce91da8cce49dba59fb1865beb961c-00000000000000000000000000000000
I20260811 11:50:40.872246 14062 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04: Tablet Copy: opened 0 blocks and 1 log segments
I20260811 11:50:40.874300 14775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260811 11:50:40.876402 14775 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c: tablet copy: Starting download of 0 data blocks...
I20260811 11:50:40.876612 14775 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c: tablet copy: Starting download of 1 WAL segments...
I20260811 11:50:40.878430 14775 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260811 11:50:40.880128 14062 tablet_copy_service.cc:342] P 9b2b536c69864468947c78d7f24fea04: Request end of tablet copy session 30ce91da8cce49dba59fb1865beb961c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.221.125@KRBTEST.COM'} at 127.0.0.1:49966
I20260811 11:50:40.880261 14062 tablet_copy_service.cc:433] P 9b2b536c69864468947c78d7f24fea04: ending tablet copy session 30ce91da8cce49dba59fb1865beb961c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 30ce91da8cce49dba59fb1865beb961c
I20260811 11:50:40.888001 14775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:40.890257 14948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:40.891624 14946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:40.891896 14945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:40.894201 14775 server_base.cc:1034] running on GCE node
I20260811 11:50:40.894385 14775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:40.894729 14775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:40.895861 14775 hybrid_clock.cc:648] HybridClock initialized: now 1786449040895852 us; error 27 us; skew 500 ppm
Aug 11 11:50:40 dist-test-slave-bndk krb5kdc[13968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449040, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:40.897310 14775 init.cc:377] Logged in from keytab as oryx/127.3.221.125@KRBTEST.COM (short username oryx)
I20260811 11:50:40.897784 14775 webserver.cc:460] Webserver started at http://127.3.221.125:40495/ using document root <none> and password file <none>
I20260811 11:50:40.897980 14775 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:40.898020 14775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:40.899195 14775 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:40.900099 14958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:40.900321 14775 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:40.900393 14775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/wal
uuid: "30ce91da8cce49dba59fb1865beb961c"
format_stamp: "Formatted at 2026-08-11 11:50:40 on dist-test-slave-bndk"
server_key: "7fb6a27c676d25b0eac064e4609d57c1"
server_key_iv: "3a6e6b90c41dc91ccfb25b18b670652b"
server_key_version: "encryptionkey@0"
I20260811 11:50:40.900480 14775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:40.918726 14775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:40.920534 14775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:40.925128 14775 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.125:39327
I20260811 11:50:40.925374 15039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.125:39327 every 8 connection(s)
I20260811 11:50:40.925727 14775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-1/data/info.pb
I20260811 11:50:40.927223 15040 sys_catalog.cc:263] Verifying existing consensus state
I20260811 11:50:40.928225 15040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c: Bootstrap starting.
I20260811 11:50:40.937330 15040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:50:40.937650 15040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c: Bootstrap complete.
I20260811 11:50:40.938175 15040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: NON_VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: true } }
I20260811 11:50:40.938297 15040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 30ce91da8cce49dba59fb1865beb961c, State: Initialized, Role: LEARNER
I20260811 11:50:40.938396 15040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: NON_VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: true } }
I20260811 11:50:40.938737 15044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: NON_VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: true } } }
I20260811 11:50:40.938885 15040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:50:40.938913 15044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [sys.catalog]: This master's current role is: LEARNER
I20260811 11:50:40.941797 15062 catalog_manager.cc:1259] Loaded cluster ID: 93c1574656b04143adc85e20ebe4d8ad
I20260811 11:50:40.941939 15062 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c: loading cluster ID for follower catalog manager: success
I20260811 11:50:40.943636 15062 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c: acquiring CA information for follower catalog manager: success
I20260811 11:50:40.944737 15062 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260811 11:50:41.020761 14944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:41.021066 14944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:41.021108 14944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:41.025578 14944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:50:41.025650 14944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:41.025669 14944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:41.025684 14944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:50:41.025698 14944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:50:41.031363 14944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:38277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:46443,127.3.221.125:39327,127.3.221.124:45801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.124.keytab
--principal=oryx/127.3.221.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.124:45801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.221.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:41.032500 14944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:41.033636 14944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:41.042568 15074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:41.042639 15075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:41.042644 14944 server_base.cc:1034] running on GCE node
W20260811 11:50:41.042568 15077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:41.043468 14944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:41.044658 14944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:41.045845 14944 hybrid_clock.cc:648] HybridClock initialized: now 1786449041045824 us; error 39 us; skew 500 ppm
Aug 11 11:50:41 dist-test-slave-bndk krb5kdc[13968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449041, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:41.048866 14944 init.cc:377] Logged in from keytab as oryx/127.3.221.124@KRBTEST.COM (short username oryx)
I20260811 11:50:41.049891 14944 webserver.cc:460] Webserver started at http://127.3.221.124:41561/ using document root <none> and password file <none>
I20260811 11:50:41.050509 14944 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:41.050573 14944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:41.050752 14944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:41.053277 14944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/data/instance:
uuid: "5ec79f33b453463abf6963e63ced5ac3"
format_stamp: "Formatted at 2026-08-11 11:50:41 on dist-test-slave-bndk"
server_key: "8b86f383023cbf729f72296c865bb3ca"
server_key_iv: "0f9ba21db7b0355e0dccae013b13e0c5"
server_key_version: "encryptionkey@0"
I20260811 11:50:41.053989 14944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/wal/instance:
uuid: "5ec79f33b453463abf6963e63ced5ac3"
format_stamp: "Formatted at 2026-08-11 11:50:41 on dist-test-slave-bndk"
server_key: "8b86f383023cbf729f72296c865bb3ca"
server_key_iv: "0f9ba21db7b0355e0dccae013b13e0c5"
server_key_version: "encryptionkey@0"
I20260811 11:50:41.058811 14944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 11:50:41.061476 15090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:41.062830 14944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:50:41.062948 14944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/wal
uuid: "5ec79f33b453463abf6963e63ced5ac3"
format_stamp: "Formatted at 2026-08-11 11:50:41 on dist-test-slave-bndk"
server_key: "8b86f383023cbf729f72296c865bb3ca"
server_key_iv: "0f9ba21db7b0355e0dccae013b13e0c5"
server_key_version: "encryptionkey@0"
I20260811 11:50:41.063045 14944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:41.075958 14325 heartbeater.cc:503] Master 127.3.221.126:46443 was elected leader, sending a full tablet report...
I20260811 11:50:41.085382 14944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:41.088538 14944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:41.088732 14944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:41.096014 14944 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.124:45801
I20260811 11:50:41.096293 15165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.124:45801 every 8 connection(s)
I20260811 11:50:41.098186 14944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/data/info.pb
I20260811 11:50:41.102609 15167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:41.106495 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 14944
I20260811 11:50:41.106621 15167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:50:41 dist-test-slave-bndk krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449041, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.124@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM
Aug 11 11:50:41 dist-test-slave-bndk krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.221.124@KRBTEST.COM
I20260811 11:50:41.120874 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.108828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49970 (local address 127.3.221.126:46443)
0811 11:50:41.108969 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.108974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.109182 (+ 208us) server_negotiation.cc:407] Connection header received
0811 11:50:41.110327 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.110331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.110396 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.110504 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:41.111750 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.112545 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.114116 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.114440 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.114535 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.116939 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:41.116944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:41.116946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:41.116981 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:41.118518 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.119051 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.119055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.119057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.119144 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.119544 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.119547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.119549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.119820 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:41.119926 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.120504 (+ 578us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.120648 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 11:50:41.121958 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.108585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.0.0.1:49970)
0811 11:50:41.109012 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.109066 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:41.110084 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:41.110702 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:41.110709 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:41.111168 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:41.111565 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.111582 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.112696 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.112701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.113971 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.113984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.114590 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.114595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.114705 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.115159 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:41.115176 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:41.116758 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:41.118627 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.118635 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.118648 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.118955 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.119228 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.119231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.119233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.119432 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.119923 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:41.119930 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:41.120249 (+ 319us) client_negotiation.cc:769] Sending connection context
0811 11:50:41.120427 (+ 178us) client_negotiation.cc:240] Negotiation successful
0811 11:50:41.120766 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":37,"thread_start_us":144,"threads_started":1}
I20260811 11:50:41.121963 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.108244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.3.221.124:45801)
0811 11:50:41.109538 (+ 1294us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.109549 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.109589 (+ 40us) server_negotiation.cc:407] Connection header received
0811 11:50:41.110089 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.110115 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.110360 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.110533 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:41.111391 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.112660 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.113509 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.114061 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.114223 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.117256 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:41.117265 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:41.117289 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:41.117318 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:41.119205 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.119712 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.119717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.119719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.119792 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.120082 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.120086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.120088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.120426 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:41.121419 (+ 993us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.121542 (+ 123us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.121711 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":1167,"thread_start_us":159,"threads_started":1}
I20260811 11:50:41.123356 15167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:50:41 dist-test-slave-bndk krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449041, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.124@KRBTEST.COM for oryx/127.3.221.125@KRBTEST.COM
I20260811 11:50:41.132189 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.123916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:39327 (local address 127.0.0.1:47592)
0811 11:50:41.124278 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.124291 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:41.124353 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:41.124732 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:41.124735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:41.124897 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:41.124988 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.124994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.125957 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.125960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.126497 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.126506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.126941 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.126945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.127041 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.127475 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:41.127489 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:41.128933 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:41.130692 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.130695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.130697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.130892 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.131156 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.131159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.131162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.131255 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.131626 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:41.131629 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:41.131726 (+ 97us) client_negotiation.cc:769] Sending connection context
0811 11:50:41.131845 (+ 119us) client_negotiation.cc:240] Negotiation successful
0811 11:50:41.132016 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292}
I20260811 11:50:41.132231 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.124029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47592 (local address 127.3.221.125:39327)
0811 11:50:41.124512 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.124516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.124526 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:50:41.124560 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.124562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.124598 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.124693 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:41.125107 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.125851 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.126606 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.126842 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.126937 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.129073 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:41.129077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:41.129080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:41.129107 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:41.130577 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.130983 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.130986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.130988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.131071 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.131337 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.131340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.131342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.131534 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:41.131624 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.131918 (+ 294us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.132091 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":123,"threads_started":1}
I20260811 11:50:41.133041 15167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:50:41 dist-test-slave-bndk krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449041, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.124@KRBTEST.COM for oryx/127.3.221.124@KRBTEST.COM
I20260811 11:50:41.141888 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.133651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:45801 (local address 127.0.0.1:38732)
0811 11:50:41.133759 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.133772 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:41.133842 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:41.134055 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:41.134056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:41.134193 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:41.134268 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.134272 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.135349 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.135352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.135843 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.135851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.136417 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.136420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.136475 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.136921 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:41.136934 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:41.138498 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:41.140458 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.140461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.140462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.140650 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.140953 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.140955 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.140956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.141024 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.141466 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:41.141468 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:41.141553 (+ 85us) client_negotiation.cc:769] Sending connection context
0811 11:50:41.141632 (+ 79us) client_negotiation.cc:240] Negotiation successful
0811 11:50:41.141752 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260811 11:50:41.141921 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.133687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38732 (local address 127.3.221.124:45801)
0811 11:50:41.133849 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.133853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.133891 (+ 38us) server_negotiation.cc:407] Connection header received
0811 11:50:41.133926 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.133930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.133974 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.134045 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:41.134367 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.135209 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.135956 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.136326 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.136423 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.138664 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:41.138669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:41.138673 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:41.138702 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:41.140326 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.140774 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.140777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.140779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.140875 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.141096 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.141098 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.141100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.141359 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:41.141467 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.141699 (+ 232us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.141813 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260811 11:50:41.145581 15167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3: Bootstrap starting.
I20260811 11:50:41.147995 15167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:41.148748 15167 log.cc:824] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:41.150712 15167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3: No bootstrap required, opened a new log
I20260811 11:50:41.153743 15167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } }
I20260811 11:50:41.153962 15167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:41.154018 15167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec79f33b453463abf6963e63ced5ac3, State: Initialized, Role: FOLLOWER
I20260811 11:50:41.154439 15167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } }
I20260811 11:50:41.154919 15182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } } }
I20260811 11:50:41.155054 15182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:50:41.155753 15167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [sys.catalog]: configured and running, proceeding with master startup.
W20260811 11:50:41.162312 15203 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:50:41.162422 15203 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:50:41.164052 14944 master_runner.cc:428] Detected that this master 5ec79f33b453463abf6963e63ced5ac3 is joining an existing cluster
I20260811 11:50:41.164106 14944 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.221.124:45801
I20260811 11:50:41.175923 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 14126
I20260811 11:50:41.176101 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.167729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47600 (local address 127.3.221.125:39327)
0811 11:50:41.167850 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.167854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.167865 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:50:41.167898 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.167901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.167938 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.168009 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:41.168657 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.169462 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.170594 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.170962 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.171041 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.172592 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:41.172596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:41.172599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:41.172624 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:41.174214 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.174702 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.174706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.174708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.174812 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.175134 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.175138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.175140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.175366 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:41.175434 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.175791 (+ 357us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.175947 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 11:50:41.176663 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.166816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:39327 (local address 127.0.0.1:47600)
0811 11:50:41.167146 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.167159 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:41.167241 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:41.168142 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:41.168145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:41.168369 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:41.168524 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.168532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.169593 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.169596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.170462 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.170473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.171210 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.171214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.171317 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.171829 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:41.171843 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:41.172333 (+ 490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:41.174336 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.174339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.174342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.174587 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.174932 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.174935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.174936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.175028 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.175502 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:41.175505 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:41.175601 (+ 96us) client_negotiation.cc:769] Sending connection context
0811 11:50:41.176310 (+ 709us) client_negotiation.cc:240] Negotiation successful
0811 11:50:41.176462 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":95,"threads_started":1}
I20260811 11:50:41.177666 15210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.166365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.0.0.1:49978)
0811 11:50:41.166657 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.166672 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:41.166774 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:41.168632 (+ 1858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:41.168634 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:41.168842 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:41.168954 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.168961 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.169976 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.169979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.170971 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.170980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.171976 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.171979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.172093 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.172576 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:41.172590 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:41.173244 (+ 654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:41.175366 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.175368 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.175370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.175589 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.175887 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.175890 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.175892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.175974 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.177126 (+ 1152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:41.177128 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:41.177255 (+ 127us) client_negotiation.cc:769] Sending connection context
0811 11:50:41.177364 (+ 109us) client_negotiation.cc:240] Negotiation successful
0811 11:50:41.177498 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":93,"threads_started":1}
I20260811 11:50:41.178208 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.167536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49978 (local address 127.3.221.126:46443)
0811 11:50:41.167822 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.167826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.167841 (+ 15us) server_negotiation.cc:407] Connection header received
0811 11:50:41.167888 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.167891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.167938 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.168009 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:41.169092 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.169846 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.171094 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.171427 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.171810 (+ 383us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.173642 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:41.173646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:41.173649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:41.173681 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:41.175230 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.175696 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.175699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.175701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.175791 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.176426 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.176429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.176431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.176646 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:41.177056 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.177989 (+ 933us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.178100 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260811 11:50:41.198892 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:45063
--local_ip_for_outbound_sockets=127.3.221.65
--tserver_master_addrs=127.3.221.126:46443
--webserver_port=41587
--webserver_interface=127.3.221.65
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.65.keytab
--principal=oryx/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:38277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.221.126:46443,127.3.221.125:39327
--tserver_master_addrs=127.3.221.126:46443,127.3.221.125:39327,127.3.221.124:45801 with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 11:50:41 dist-test-slave-bndk krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449039, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.126@KRBTEST.COM for oryx/127.3.221.124@KRBTEST.COM
I20260811 11:50:41.208614 15212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.167235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:45801 (local address 127.0.0.1:38742)
0811 11:50:41.167461 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.167474 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:41.167552 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:41.171560 (+ 4008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:41.171563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:41.201446 (+ 29883us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:41.201597 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.201606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.202605 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.202608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.203450 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.203463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.204080 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.204083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.204167 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.204579 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:41.204594 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:41.205106 (+ 512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:41.207130 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.207133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.207135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.207362 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.207611 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.207614 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.207616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.207688 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.208092 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:41.208094 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:41.208168 (+ 74us) client_negotiation.cc:769] Sending connection context
0811 11:50:41.208255 (+ 87us) client_negotiation.cc:240] Negotiation successful
0811 11:50:41.208404 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":105,"threads_started":1}
I20260811 11:50:41.209116 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.168178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38742 (local address 127.3.221.124:45801)
0811 11:50:41.168348 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.168352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.168364 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:50:41.168395 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.168397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.168431 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.168515 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:41.201712 (+ 33197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.202481 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.203579 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.203888 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.203967 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.205291 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:41.205295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:41.205298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:41.205323 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:41.207010 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.207455 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.207458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.207460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.207532 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.207775 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.207778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.207780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.207985 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:41.208063 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.208865 (+ 802us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.208978 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260811 11:50:41.213640 14920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.181852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:45801 (local address 127.0.0.1:38746)
0811 11:50:41.182069 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.182088 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:41.182179 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:41.197984 (+ 15805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:41.197989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:41.198226 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:41.198371 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.198379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.199648 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.199652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.200551 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.200567 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.201286 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.201290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.201395 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.202043 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:41.202059 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:41.203623 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:41.211796 (+ 8173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.211800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.211803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.212086 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.212401 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.212405 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.212407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.212504 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.212965 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:41.212968 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:41.213076 (+ 108us) client_negotiation.cc:769] Sending connection context
0811 11:50:41.213159 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 11:50:41.213468 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260811 11:50:41.214366 15214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.190872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.3.221.124:45801)
0811 11:50:41.197533 (+ 6661us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.197542 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.197566 (+ 24us) server_negotiation.cc:407] Connection header received
0811 11:50:41.197678 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.197682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.197763 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.198501 (+ 738us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:41.198649 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.199497 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.200743 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.201099 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.203779 (+ 2680us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.203910 (+ 131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:41.203914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:41.203917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:41.203934 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:41.211632 (+ 7698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.212208 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.212211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.212213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.212296 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.212610 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.212613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.212615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.212849 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:41.213991 (+ 1142us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.214069 (+ 78us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.214195 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6540,"thread_start_us":112,"threads_started":1}
I20260811 11:50:41.215777 14034 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.221.124:45801: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3"
member_type: NON_VOTER
last_known_addr {
host: "127.3.221.124"
port: 45801
}
attrs {
promote: true
}
}
dest_uuid: "9b2b536c69864468947c78d7f24fea04"
cas_config_opid_index: 5
I20260811 11:50:41.216455 14034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: NON_VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: true } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: NON_VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: true } }
I20260811 11:50:41.217700 14922 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5ec79f33b453463abf6963e63ced5ac3 (127.3.221.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: NON_VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: true } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: NON_VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: true } } }
I20260811 11:50:41.219270 14922 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.221.124:45801
I20260811 11:50:41.219468 14922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9b2b536c69864468947c78d7f24fea04" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: NON_VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: true } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: NON_VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: true } } }
I20260811 11:50:41.219573 14922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:41.220316 14944 master.cc:537] Master@127.3.221.124:45801 shutting down...
I20260811 11:50:41.220707 15130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:41.222184 15130 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 1 FOLLOWER]: Refusing update from remote peer 9b2b536c69864468947c78d7f24fea04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260811 11:50:41.223902 14920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.217766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:39327 (local address 127.0.0.1:47616)
0811 11:50:41.218229 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.218242 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:41.218341 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:41.218757 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:41.218780 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:41.218797 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:41.218889 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.218896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.220911 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.220914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.222238 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.222250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.223452 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.223456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.223555 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.223609 (+ 54us) client_negotiation.cc:769] Sending connection context
0811 11:50:41.223679 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 11:50:41.223764 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387}
I20260811 11:50:41.223932 14924 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: NON_VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:41.224396 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.218012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47616 (local address 127.3.221.125:39327)
0811 11:50:41.218134 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.218139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.218422 (+ 283us) server_negotiation.cc:407] Connection header received
0811 11:50:41.218480 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.218483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.218545 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.218631 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:41.219093 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.219906 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.222405 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.223088 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.223196 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.223383 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.224171 (+ 788us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.224262 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 11:50:41.224884 14997 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 1 LEARNER]: Refusing update from remote peer 9b2b536c69864468947c78d7f24fea04: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260811 11:50:41.225984 14924 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: NON_VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260811 11:50:41.228199 14997 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5ec79f33b453463abf6963e63ced5ac3 (127.3.221.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: NON_VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: true } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: NON_VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: true } } }
W20260811 11:50:41.230827 14015 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 -> Peer 5ec79f33b453463abf6963e63ced5ac3 (127.3.221.124:45801): Couldn't send request to peer 5ec79f33b453463abf6963e63ced5ac3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260811 11:50:41.231834 15044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b2b536c69864468947c78d7f24fea04. Latest consensus state: current_term: 1 leader_uuid: "9b2b536c69864468947c78d7f24fea04" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: NON_VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: true } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: NON_VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: true } } }
I20260811 11:50:41.231967 15044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [sys.catalog]: This master's current role is: LEARNER
I20260811 11:50:41.232064 15044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b2b536c69864468947c78d7f24fea04" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: NON_VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: true } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: NON_VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: true } } }
I20260811 11:50:41.232134 15044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [sys.catalog]: This master's current role is: LEARNER
I20260811 11:50:41.232759 14944 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:41.233080 14944 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:41.233199 14944 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3: stopping tablet replica
I20260811 11:50:41.239347 14924 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04: attempting to promote NON_VOTER 30ce91da8cce49dba59fb1865beb961c to VOTER
I20260811 11:50:41.239904 14924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: NON_VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: true } }
I20260811 11:50:41.241474 14920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.241044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:45801 (local address 127.0.0.1:38762)
0811 11:50:41.241325 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.241387 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:45801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
I20260811 11:50:41.242570 14997 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 1 LEARNER]: Refusing update from remote peer 9b2b536c69864468947c78d7f24fea04: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260811 11:50:41.243152 14922 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260811 11:50:41.243880 14920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.243596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:45801 (local address 127.0.0.1:38764)
0811 11:50:41.243735 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.243805 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:45801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260811 11:50:41.244252 14015 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 -> Peer 5ec79f33b453463abf6963e63ced5ac3 (127.3.221.124:45801): Couldn't send request to peer 5ec79f33b453463abf6963e63ced5ac3. Status: Network error: Client connection negotiation failed: client connection to 127.3.221.124:45801: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260811 11:50:41.245772 14922 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 30ce91da8cce49dba59fb1865beb961c (127.3.221.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: NON_VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: true } } }
I20260811 11:50:41.247346 14922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9b2b536c69864468947c78d7f24fea04" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: NON_VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: true } } }
I20260811 11:50:41.247462 14922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:41.247579 14922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b2b536c69864468947c78d7f24fea04" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: NON_VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: true } } }
I20260811 11:50:41.247654 14922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:41.248442 14997 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 30ce91da8cce49dba59fb1865beb961c (127.3.221.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: NON_VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: true } } }
I20260811 11:50:41.250787 15219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b2b536c69864468947c78d7f24fea04" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: NON_VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: true } } }
I20260811 11:50:41.250922 15219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:50:41.251576 14944 master.cc:552] Master@127.3.221.124:45801 shutdown complete.
I20260811 11:50:41.251633 14944 master_runner.cc:305] Clearing existing system tablet
I20260811 11:50:41.251884 14944 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:41.251940 14944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:41.253544 14944 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:41.257422 15224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:41.257682 14944 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:41.257810 14944 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260811 11:50:41.257851 14944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/wal
uuid: "5ec79f33b453463abf6963e63ced5ac3"
format_stamp: "Formatted at 2026-08-11 11:50:41 on dist-test-slave-bndk"
server_key: "8b86f383023cbf729f72296c865bb3ca"
server_key_iv: "0f9ba21db7b0355e0dccae013b13e0c5"
server_key_version: "encryptionkey@0"
I20260811 11:50:41.260030 14944 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:41.262528 14944 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260811 11:50:41.262629 14944 log.cc:1196] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260811 11:50:41.262887 14944 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3: Deleting consensus metadata
I20260811 11:50:41.263057 14944 master_runner.cc:315] Copying system tablet from 127.3.221.126:46443
I20260811 11:50:41.264238 14944 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3: tablet copy: Beginning tablet copy session from remote peer at address 127.3.221.126:46443
I20260811 11:50:41.280117 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.269951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49986 (local address 127.3.221.126:46443)
0811 11:50:41.270130 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.270135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.270565 (+ 430us) server_negotiation.cc:407] Connection header received
0811 11:50:41.270764 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.270768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.270817 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.270867 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:41.271556 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.272390 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.274014 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.274401 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.274478 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.276506 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:41.276510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:41.276513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:41.276543 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:41.278235 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.278737 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.278741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.278743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.278830 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.279124 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.279127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.279129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.279352 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:41.279452 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.279804 (+ 352us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.279939 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 11:50:41.280753 15237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.270135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.0.0.1:49986)
0811 11:50:41.270479 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.270494 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:41.270653 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:41.271020 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:41.271023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:41.271278 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:41.271415 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.271423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.272679 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.272682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.273877 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.273890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.274655 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.274658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.274751 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.275857 (+ 1106us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:41.275873 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:41.276319 (+ 446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:41.278360 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.278363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.278366 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.278625 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.278938 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.278941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.278943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.279028 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.279525 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:41.279528 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:41.279628 (+ 100us) client_negotiation.cc:769] Sending connection context
0811 11:50:41.280357 (+ 729us) client_negotiation.cc:240] Negotiation successful
0811 11:50:41.280578 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":68,"threads_started":1}
I20260811 11:50:41.281196 14062 tablet_copy_service.cc:140] P 9b2b536c69864468947c78d7f24fea04: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5ec79f33b453463abf6963e63ced5ac3 ({username='oryx', principal='oryx/127.3.221.124@KRBTEST.COM'} at 127.0.0.1:49986)
I20260811 11:50:41.281337 14062 tablet_copy_service.cc:161] P 9b2b536c69864468947c78d7f24fea04: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5ec79f33b453463abf6963e63ced5ac3 at {username='oryx', principal='oryx/127.3.221.124@KRBTEST.COM'} at 127.0.0.1:49986: session id = 5ec79f33b453463abf6963e63ced5ac3-00000000000000000000000000000000
I20260811 11:50:41.282470 14062 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04: Tablet Copy: opened 0 blocks and 1 log segments
I20260811 11:50:41.283953 14944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260811 11:50:41.286479 14944 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3: tablet copy: Starting download of 0 data blocks...
I20260811 11:50:41.286695 14944 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3: tablet copy: Starting download of 1 WAL segments...
I20260811 11:50:41.288208 14944 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260811 11:50:41.290001 14062 tablet_copy_service.cc:342] P 9b2b536c69864468947c78d7f24fea04: Request end of tablet copy session 5ec79f33b453463abf6963e63ced5ac3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.221.124@KRBTEST.COM'} at 127.0.0.1:49986
I20260811 11:50:41.290093 14062 tablet_copy_service.cc:433] P 9b2b536c69864468947c78d7f24fea04: ending tablet copy session 5ec79f33b453463abf6963e63ced5ac3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5ec79f33b453463abf6963e63ced5ac3
I20260811 11:50:41.299154 14944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:41.302488 15241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:41.305477 14537 heartbeater.cc:503] Master 127.3.221.126:46443 was elected leader, sending a full tablet report...
W20260811 11:50:41.308099 15240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:41.308238 15246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:41.308336 14944 server_base.cc:1034] running on GCE node
I20260811 11:50:41.308485 14944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:41.308799 14944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:41.309938 14944 hybrid_clock.cc:648] HybridClock initialized: now 1786449041309923 us; error 31 us; skew 500 ppm
Aug 11 11:50:41 dist-test-slave-bndk krb5kdc[13968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449041, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:41.311921 14944 init.cc:377] Logged in from keytab as oryx/127.3.221.124@KRBTEST.COM (short username oryx)
I20260811 11:50:41.312430 14944 webserver.cc:460] Webserver started at http://127.3.221.124:43213/ using document root <none> and password file <none>
I20260811 11:50:41.312623 14944 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:41.312659 14944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:41.314276 14944 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:41.315481 15261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:41.315734 14944 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:41.315799 14944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/wal
uuid: "5ec79f33b453463abf6963e63ced5ac3"
format_stamp: "Formatted at 2026-08-11 11:50:41 on dist-test-slave-bndk"
server_key: "8b86f383023cbf729f72296c865bb3ca"
server_key_iv: "0f9ba21db7b0355e0dccae013b13e0c5"
server_key_version: "encryptionkey@0"
I20260811 11:50:41.315892 14944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:41.343483 14944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:41.345631 14944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:41.350607 14944 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.124:45801
I20260811 11:50:41.350646 15345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.124:45801 every 8 connection(s)
I20260811 11:50:41.351257 14944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/master-2/data/info.pb
I20260811 11:50:41.353142 15346 sys_catalog.cc:263] Verifying existing consensus state
I20260811 11:50:41.354254 15346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3: Bootstrap starting.
I20260811 11:50:41.366032 15346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:50:41.366376 15346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3: Bootstrap complete.
I20260811 11:50:41.366993 15346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: NON_VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: true } }
I20260811 11:50:41.367143 15346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5ec79f33b453463abf6963e63ced5ac3, State: Initialized, Role: LEARNER
I20260811 11:50:41.367255 15346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: NON_VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: true } }
I20260811 11:50:41.367627 15352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: NON_VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: true } } }
I20260811 11:50:41.367739 15352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [sys.catalog]: This master's current role is: LEARNER
I20260811 11:50:41.367774 15346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:50:41.371598 15372 catalog_manager.cc:1259] Loaded cluster ID: 93c1574656b04143adc85e20ebe4d8ad
I20260811 11:50:41.371652 15372 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3: loading cluster ID for follower catalog manager: success
I20260811 11:50:41.373354 15372 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3: acquiring CA information for follower catalog manager: success
I20260811 11:50:41.374524 15372 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260811 11:50:41.376137 15215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:41.376391 15215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:41.376443 15215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:41.380615 15215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:41.380685 15215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:41.380750 15215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:50:41.386407 15215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:38277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.65.keytab
--principal=oryx/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:45063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=41587
--tserver_master_addrs=127.3.221.126:46443,127.3.221.125:39327,127.3.221.124:45801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:41.387564 15215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:41.388630 15215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:41.398945 15383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:41.399138 15215 server_base.cc:1034] running on GCE node
W20260811 11:50:41.399022 15389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:41.398944 15384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:41.399777 15215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:41.400628 15215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:41.401793 15215 hybrid_clock.cc:648] HybridClock initialized: now 1786449041401774 us; error 34 us; skew 500 ppm
Aug 11 11:50:41 dist-test-slave-bndk krb5kdc[13968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449041, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:41.404697 15215 init.cc:377] Logged in from keytab as oryx/127.3.221.65@KRBTEST.COM (short username oryx)
I20260811 11:50:41.405733 15215 webserver.cc:460] Webserver started at http://127.3.221.65:41587/ using document root <none> and password file <none>
I20260811 11:50:41.406299 15215 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:41.406406 15215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:41.410492 15215 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260811 11:50:41.412798 15402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:41.414302 15215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 11:50:41.414456 15215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "6065b7065aa84491a806b7276eb7b333"
format_stamp: "Formatted at 2026-08-11 11:50:40 on dist-test-slave-bndk"
server_key: "99c97e09a7706844e2c3032d92583c84"
server_key_iv: "a1f7b6b6bedd5dd4e5d1183b62f320be"
server_key_version: "encryptionkey@0"
I20260811 11:50:41.415028 15215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:41.430133 15215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:41.433513 15215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:41.433701 15215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:41.434335 15215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:41.435246 15215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:41.435305 15215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:41.435362 15215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:41.435388 15215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:41.445787 15215 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:45063
I20260811 11:50:41.445824 15590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:45063 every 8 connection(s)
I20260811 11:50:41.446895 15215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:50:41.452993 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 15215
I20260811 11:50:41.453238 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 14332
Aug 11 11:50:41 dist-test-slave-bndk krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449041, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.124@KRBTEST.COM
Aug 11 11:50:41 dist-test-slave-bndk krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449041, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.125@KRBTEST.COM
Aug 11 11:50:41 dist-test-slave-bndk krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449041, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM
I20260811 11:50:41.485406 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.449564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:50445 (local address 127.3.221.124:45801)
0811 11:50:41.450556 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.450561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.450609 (+ 48us) server_negotiation.cc:407] Connection header received
0811 11:50:41.451248 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.451252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.451311 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.451393 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:41.452976 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.473619 (+ 20643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.475452 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.475988 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.476155 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.479477 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:41.479481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:41.479484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:41.479518 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:41.481434 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.482176 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.482181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.482183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.482282 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.482700 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.482704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.482706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.482966 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:41.483058 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.485041 (+ 1983us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.485192 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":84,"threads_started":1}
I20260811 11:50:41.486037 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.449433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:57001 (local address 127.3.221.125:39327)
0811 11:50:41.449551 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.449555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.450794 (+ 1239us) server_negotiation.cc:407] Connection header received
0811 11:50:41.451269 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.451272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.451332 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.451386 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:41.452955 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.453800 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.479195 (+ 25395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.479538 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.479623 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.482553 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:41.482557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:41.482560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:41.482590 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:41.484197 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.484674 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.484678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.484680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.484772 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.485073 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.485076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.485078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.485356 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:41.485445 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.485778 (+ 333us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.485902 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 11:50:41.486733 15600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.449340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:39327 (local address 127.3.221.65:57001)
0811 11:50:41.450677 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.450691 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:41.451038 (+ 347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:41.451592 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:41.451600 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:41.452165 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:41.452781 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.452802 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.478121 (+ 25319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.478125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.478994 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.479009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.479816 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.479820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.479910 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.480346 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:41.480361 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:41.482388 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:41.484347 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.484350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.484352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.484563 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.484879 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.484881 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.484883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.484971 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.485508 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:41.485511 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:41.485602 (+ 91us) client_negotiation.cc:769] Sending connection context
0811 11:50:41.486252 (+ 650us) client_negotiation.cc:240] Negotiation successful
0811 11:50:41.486417 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"mutex_wait_us":1020,"spinlock_wait_cycles":3584,"thread_start_us":144,"threads_started":1}
I20260811 11:50:41.487689 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:40795
--local_ip_for_outbound_sockets=127.3.221.66
--tserver_master_addrs=127.3.221.126:46443
--webserver_port=39259
--webserver_interface=127.3.221.66
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.66.keytab
--principal=oryx/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:38277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.221.126:46443,127.3.221.125:39327
--tserver_master_addrs=127.3.221.126:46443,127.3.221.125:39327,127.3.221.124:45801 with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:50:41.487721 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.448815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:45183 (local address 127.3.221.126:46443)
0811 11:50:41.448972 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.448976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.450122 (+ 1146us) server_negotiation.cc:407] Connection header received
0811 11:50:41.451374 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.451377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.451430 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.451498 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:41.452956 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.453842 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.481501 (+ 27659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.481832 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.481951 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.484322 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:41.484326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:41.484329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:41.484359 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:41.485858 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.486365 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.486368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.486370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.486471 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.486788 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.486792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.486794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.487005 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:41.487087 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.487476 (+ 389us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.487589 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 11:50:41.486737 15596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.449340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:45801 (local address 127.3.221.65:50445)
0811 11:50:41.449944 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.449993 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:41.451024 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:41.451591 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:41.451598 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:41.452152 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:41.452779 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.452802 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.473896 (+ 21094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.473900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.475243 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.475261 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.476206 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.476210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.476425 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.477233 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:41.477256 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:41.479257 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:41.481619 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.481626 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.481647 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.482044 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.482388 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.482391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.482393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.482596 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.483121 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:41.483130 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:41.483503 (+ 373us) client_negotiation.cc:769] Sending connection context
0811 11:50:41.483744 (+ 241us) client_negotiation.cc:240] Negotiation successful
0811 11:50:41.484409 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":40,"thread_start_us":163,"threads_started":1}
I20260811 11:50:41.488116 15594 heartbeater.cc:347] Connected to a master server at 127.3.221.125:39327
I20260811 11:50:41.488317 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.449340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.65:45183)
0811 11:50:41.449940 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.449993 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:41.451025 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:41.451603 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:41.451606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:41.452172 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:41.452781 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.452802 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.480185 (+ 27383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.480189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.481059 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.481073 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.481991 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.481995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.482176 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.482683 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:41.482697 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:41.484115 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:41.485991 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.485994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.485995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.486226 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.486597 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.486600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.486602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.486689 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.487132 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:41.487135 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:41.487230 (+ 95us) client_negotiation.cc:769] Sending connection context
0811 11:50:41.487331 (+ 101us) client_negotiation.cc:240] Negotiation successful
0811 11:50:41.487465 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":137,"spinlock_wait_cycles":46464,"thread_start_us":157,"threads_started":1}
I20260811 11:50:41.488379 15594 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:41.488885 15592 heartbeater.cc:347] Connected to a master server at 127.3.221.124:45801
I20260811 11:50:41.488974 15592 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:41.489159 15594 heartbeater.cc:511] Master 127.3.221.125:39327 requested a full tablet report, sending...
I20260811 11:50:41.489166 15592 heartbeater.cc:511] Master 127.3.221.124:45801 requested a full tablet report, sending...
I20260811 11:50:41.489713 15593 heartbeater.cc:347] Connected to a master server at 127.3.221.126:46443
I20260811 11:50:41.489804 15593 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:41.489970 15593 heartbeater.cc:511] Master 127.3.221.126:46443 requested a full tablet report, sending...
I20260811 11:50:41.490731 14034 ts_manager.cc:194] Re-registered known tserver with Master: 6065b7065aa84491a806b7276eb7b333 (127.3.221.65:45063)
I20260811 11:50:41.490898 14983 ts_manager.cc:194] Registered new tserver with Master: 6065b7065aa84491a806b7276eb7b333 (127.3.221.65:45063)
I20260811 11:50:41.490899 15279 ts_manager.cc:194] Registered new tserver with Master: 6065b7065aa84491a806b7276eb7b333 (127.3.221.65:45063)
I20260811 11:50:41.491508 14034 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:45183
I20260811 11:50:41.530005 14769 heartbeater.cc:503] Master 127.3.221.126:46443 was elected leader, sending a full tablet report...
W20260811 11:50:41.611505 15601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:41.611766 15601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:41.611827 15601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:41.615787 15601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:41.615852 15601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:41.615931 15601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:50:41.621446 15601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:38277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.66.keytab
--principal=oryx/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:40795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=39259
--tserver_master_addrs=127.3.221.126:46443,127.3.221.125:39327,127.3.221.124:45801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:41.622568 15601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:41.623564 15601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:41.633231 15614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:41.633361 15601 server_base.cc:1034] running on GCE node
W20260811 11:50:41.633195 15615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:41.633195 15617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:41.633876 15601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:41.634527 15601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:41.635686 15601 hybrid_clock.cc:648] HybridClock initialized: now 1786449041635666 us; error 33 us; skew 500 ppm
Aug 11 11:50:41 dist-test-slave-bndk krb5kdc[13968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449041, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:41.638447 15601 init.cc:377] Logged in from keytab as oryx/127.3.221.66@KRBTEST.COM (short username oryx)
I20260811 11:50:41.639458 15601 webserver.cc:460] Webserver started at http://127.3.221.66:39259/ using document root <none> and password file <none>
I20260811 11:50:41.640060 15601 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:41.640161 15601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:41.644157 15601 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260811 11:50:41.646282 15627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:41.647639 15601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 11:50:41.647753 15601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "09017061acfe48f6bb57b06e16ce084e"
format_stamp: "Formatted at 2026-08-11 11:50:40 on dist-test-slave-bndk"
server_key: "2281dd4f670e08847f6fdcbff413ea07"
server_key_iv: "ffb82508047ddba81259adc3714279db"
server_key_version: "encryptionkey@0"
I20260811 11:50:41.648308 15601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:41.667600 15601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:41.670768 15601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:41.670934 15601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:41.671554 15601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:41.672426 15601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:41.672474 15601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:41.672516 15601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:41.672544 15601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:41.681561 15601 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:40795
I20260811 11:50:41.681586 15812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:40795 every 8 connection(s)
I20260811 11:50:41.682590 15601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:50:41.692963 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 15601
I20260811 11:50:41.693161 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 14544
Aug 11 11:50:41 dist-test-slave-bndk krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449041, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM
Aug 11 11:50:41 dist-test-slave-bndk krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449041, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.125@KRBTEST.COM
I20260811 11:50:41.706216 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.684553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:45155 (local address 127.3.221.126:46443)
0811 11:50:41.684895 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.684900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.685640 (+ 740us) server_negotiation.cc:407] Connection header received
0811 11:50:41.686690 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.686693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.686753 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.686827 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:41.688193 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.689036 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.691059 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.691495 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.691561 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.695095 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:41.695100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:41.695102 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:41.695132 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:41.702398 (+ 7266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.703139 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.703143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.703145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.703247 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.703697 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.703700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.703702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.703957 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:41.704043 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.705894 (+ 1851us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.706045 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248}
I20260811 11:50:41.706216 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.684405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:60073 (local address 127.3.221.125:39327)
0811 11:50:41.684691 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.684696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.686121 (+ 1425us) server_negotiation.cc:407] Connection header received
0811 11:50:41.686708 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.686711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.686765 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.686825 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:41.689179 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.689732 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.691063 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.691431 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.691514 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.702104 (+ 10590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:41.702108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:41.702111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:41.702142 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:41.703934 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.704405 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.704408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.704411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.704497 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.704765 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.704768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.704770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.704978 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:41.705043 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.705918 (+ 875us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.706056 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260811 11:50:41.708032 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.684937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:39327 (local address 127.3.221.66:60073)
0811 11:50:41.686004 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.686019 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:41.686493 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:41.687024 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:41.687032 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:41.687521 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:41.688060 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.688070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.689881 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.689884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.690906 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.690919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.691736 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.691741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.692011 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.692707 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:41.692735 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:41.701901 (+ 9166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:41.704061 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.704064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.704066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.704296 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.704598 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.704601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.704602 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.704679 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.705127 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:41.705135 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:41.705557 (+ 422us) client_negotiation.cc:769] Sending connection context
0811 11:50:41.706418 (+ 861us) client_negotiation.cc:240] Negotiation successful
0811 11:50:41.706829 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":981,"mutex_wait_us":254,"spinlock_wait_cycles":75776,"thread_start_us":173,"threads_started":1}
I20260811 11:50:41.708047 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.684937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.66:45155)
0811 11:50:41.685481 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.685533 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:41.686485 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:41.687023 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:41.687030 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:41.687490 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:41.688050 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.688065 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.689186 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.689189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.690913 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.690925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.691736 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.691740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.691998 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.692748 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:41.692764 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:41.694871 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:41.702563 (+ 7692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.702572 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.702596 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.703012 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.703367 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.703370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.703372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.703579 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.705184 (+ 1605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:41.705187 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:41.705557 (+ 370us) client_negotiation.cc:769] Sending connection context
0811 11:50:41.706756 (+ 1199us) client_negotiation.cc:240] Negotiation successful
0811 11:50:41.706910 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":243,"spinlock_wait_cycles":23808,"thread_start_us":167,"threads_started":1}
I20260811 11:50:41.709764 15816 heartbeater.cc:347] Connected to a master server at 127.3.221.125:39327
I20260811 11:50:41.710032 15816 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:41.710271 15814 heartbeater.cc:347] Connected to a master server at 127.3.221.126:46443
I20260811 11:50:41.710368 15814 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:41.710618 15814 heartbeater.cc:511] Master 127.3.221.126:46443 requested a full tablet report, sending...
I20260811 11:50:41.710618 15816 heartbeater.cc:511] Master 127.3.221.125:39327 requested a full tablet report, sending...
I20260811 11:50:41.711850 14034 ts_manager.cc:194] Re-registered known tserver with Master: 09017061acfe48f6bb57b06e16ce084e (127.3.221.66:40795)
I20260811 11:50:41.711879 14983 ts_manager.cc:194] Registered new tserver with Master: 09017061acfe48f6bb57b06e16ce084e (127.3.221.66:40795)
I20260811 11:50:41.712471 14034 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:45155
I20260811 11:50:41.715322 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:44329
--local_ip_for_outbound_sockets=127.3.221.67
--tserver_master_addrs=127.3.221.126:46443
--webserver_port=41303
--webserver_interface=127.3.221.67
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.67.keytab
--principal=oryx/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:38277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.221.126:46443,127.3.221.125:39327
--tserver_master_addrs=127.3.221.126:46443,127.3.221.125:39327,127.3.221.124:45801 with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 11:50:41 dist-test-slave-bndk krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449041, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.124@KRBTEST.COM
I20260811 11:50:41.720117 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.684998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:58167 (local address 127.3.221.124:45801)
0811 11:50:41.685142 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.685147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.686206 (+ 1059us) server_negotiation.cc:407] Connection header received
0811 11:50:41.686708 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.686711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.686766 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.686816 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:41.688226 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.689032 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.691059 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.691434 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.691513 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.716896 (+ 25383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:41.716901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:41.716903 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:41.716942 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:41.718314 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.718784 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.718787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.718789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.718869 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.719178 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.719182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.719184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.719380 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:41.719478 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.719797 (+ 319us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.719950 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 11:50:41.720443 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.684937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:45801 (local address 127.3.221.66:58167)
0811 11:50:41.686118 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.686130 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:41.686492 (+ 362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:41.687024 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:41.687032 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:41.687520 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:41.688064 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.688070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.689186 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.689190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.690913 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.690925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.691736 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.691740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.692006 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.715256 (+ 23250us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:41.715270 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:41.716714 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:41.718431 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.718435 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.718437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.718675 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.718954 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.718965 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.718967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.719054 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.719475 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:41.719478 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:41.719573 (+ 95us) client_negotiation.cc:769] Sending connection context
0811 11:50:41.719702 (+ 129us) client_negotiation.cc:240] Negotiation successful
0811 11:50:41.719853 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"mutex_wait_us":564,"spinlock_wait_cycles":13312,"thread_start_us":140,"threads_started":1}
I20260811 11:50:41.721145 15813 heartbeater.cc:347] Connected to a master server at 127.3.221.124:45801
I20260811 11:50:41.721306 15813 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:41.721552 15813 heartbeater.cc:511] Master 127.3.221.124:45801 requested a full tablet report, sending...
I20260811 11:50:41.722309 15279 ts_manager.cc:194] Registered new tserver with Master: 09017061acfe48f6bb57b06e16ce084e (127.3.221.66:40795)
I20260811 11:50:41.803900 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.800409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:45801 (local address 127.0.0.1:38778)
0811 11:50:41.800662 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.800678 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:41.800797 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:41.801091 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:41.801094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:41.801113 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:41.801244 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.801250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.802036 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.802039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.802818 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.802827 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.803494 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.803497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.803589 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.803612 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 11:50:41.803675 (+ 63us) client_negotiation.cc:240] Negotiation successful
0811 11:50:41.803754 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":79,"threads_started":1}
I20260811 11:50:41.803905 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.800496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38778 (local address 127.3.221.124:45801)
0811 11:50:41.800662 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.800666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.800754 (+ 88us) server_negotiation.cc:407] Connection header received
0811 11:50:41.800924 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.800926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.800982 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.801067 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:41.801351 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.801926 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.802948 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.803376 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.803474 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.803643 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.803685 (+ 42us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.803754 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 11:50:41.804885 15305 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260811 11:50:41.805364 15352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b2b536c69864468947c78d7f24fea04. Latest consensus state: current_term: 1 leader_uuid: "9b2b536c69864468947c78d7f24fea04" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: NON_VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: true } } }
I20260811 11:50:41.805531 15352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [sys.catalog]: This master's current role is: LEARNER
W20260811 11:50:41.835052 15823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:41.835346 15823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:41.835408 15823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:41.839654 15823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:41.839715 15823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:41.839798 15823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:50:41.845386 15823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:38277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.67.keytab
--principal=oryx/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:44329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=41303
--tserver_master_addrs=127.3.221.126:46443,127.3.221.125:39327,127.3.221.124:45801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:41.846524 15823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:41.847628 15823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:41.857559 15833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:41.857544 15832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:41.857656 15823 server_base.cc:1034] running on GCE node
W20260811 11:50:41.857563 15837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:41.858242 15823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:41.858892 15823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:41.860090 15823 hybrid_clock.cc:648] HybridClock initialized: now 1786449041860069 us; error 37 us; skew 500 ppm
Aug 11 11:50:41 dist-test-slave-bndk krb5kdc[13968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449041, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:41.862973 15823 init.cc:377] Logged in from keytab as oryx/127.3.221.67@KRBTEST.COM (short username oryx)
I20260811 11:50:41.863894 15823 webserver.cc:460] Webserver started at http://127.3.221.67:41303/ using document root <none> and password file <none>
I20260811 11:50:41.864461 15823 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:41.864538 15823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:41.868745 15823 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260811 11:50:41.871073 15846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:41.872440 15823 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 11:50:41.872561 15823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "777856fcd5864cffa63bd364422fc3a8"
format_stamp: "Formatted at 2026-08-11 11:50:40 on dist-test-slave-bndk"
server_key: "ff5157be589d6aaa04bcea18f2bc6033"
server_key_iv: "49ab9b01e192043427145a864320d5d9"
server_key_version: "encryptionkey@0"
I20260811 11:50:41.873126 15823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:41.893355 15823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:41.896927 15823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:41.897096 15823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:41.897859 15823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:41.898768 15823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:41.898818 15823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:41.898856 15823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:41.898878 15823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:41.908041 15823 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:44329
I20260811 11:50:41.908071 16030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:44329 every 8 connection(s)
I20260811 11:50:41.909086 15823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:50:41.913105 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 15823
Aug 11 11:50:41 dist-test-slave-bndk krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449041, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.124@KRBTEST.COM
Aug 11 11:50:41 dist-test-slave-bndk krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449041, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.125@KRBTEST.COM
Aug 11 11:50:41 dist-test-slave-bndk krb5kdc[13968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449041, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM
I20260811 11:50:41.927426 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.911182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:42057 (local address 127.3.221.124:45801)
0811 11:50:41.911501 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.911506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.912570 (+ 1064us) server_negotiation.cc:407] Connection header received
0811 11:50:41.913681 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.913685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.913744 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.913794 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:41.915855 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.916702 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.918386 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.918698 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.918784 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.922989 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:41.922993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:41.922996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:41.923024 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:41.924897 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.925625 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.925628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.925631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.925722 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.926194 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.926198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.926199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.926419 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:41.926519 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.927157 (+ 638us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.927273 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260811 11:50:41.928653 16044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.911538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:45801 (local address 127.3.221.67:42057)
0811 11:50:41.912468 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.912481 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:41.913424 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:41.914218 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:41.914228 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:41.914893 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:41.915681 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.915702 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.917153 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.917156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.918267 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.918278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.918836 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.918840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.919108 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.919847 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:41.919879 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:41.922839 (+ 2960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:41.925035 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.925042 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.925061 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.925477 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.925830 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.925833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.925835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.926096 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.926549 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:41.926556 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:41.926887 (+ 331us) client_negotiation.cc:769] Sending connection context
0811 11:50:41.927099 (+ 212us) client_negotiation.cc:240] Negotiation successful
0811 11:50:41.927456 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":326,"spinlock_wait_cycles":48896,"thread_start_us":139,"threads_started":1}
I20260811 11:50:41.929381 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.911767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:47231 (local address 127.3.221.126:46443)
0811 11:50:41.911897 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.911901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.912299 (+ 398us) server_negotiation.cc:407] Connection header received
0811 11:50:41.913689 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.913693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.913752 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.913804 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:41.915860 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.916703 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.919161 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.919613 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.919718 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.923934 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:41.923939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:41.923942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:41.923973 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:41.925679 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.927770 (+ 2091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.927774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.927776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.927886 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.928186 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.928190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.928192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.928400 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:41.928482 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.929084 (+ 602us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.929239 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 11:50:41.930024 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.911539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.67:47231)
0811 11:50:41.912143 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.912197 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:41.913436 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:41.914219 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:41.914228 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:41.914980 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:41.915681 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.915702 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.916918 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.916921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.918267 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.918278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.919933 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.919936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.920024 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.921352 (+ 1328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:41.921367 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:41.923425 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:41.927177 (+ 3752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.927180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.927182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.927414 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.928000 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.928002 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.928004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.928088 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.928539 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:41.928542 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:41.928631 (+ 89us) client_negotiation.cc:769] Sending connection context
0811 11:50:41.929697 (+ 1066us) client_negotiation.cc:240] Negotiation successful
0811 11:50:41.929860 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":188,"spinlock_wait_cycles":11136,"thread_start_us":145,"threads_started":1}
I20260811 11:50:41.930121 16033 heartbeater.cc:347] Connected to a master server at 127.3.221.124:45801
I20260811 11:50:41.930414 16054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.915646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38782 (local address 127.3.221.124:45801)
0811 11:50:41.920361 (+ 4715us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.920365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.920377 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:50:41.920412 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.920415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.920461 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.920545 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:41.921518 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.922218 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.923366 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.923685 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.923779 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.925669 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:41.925673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:41.925676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:41.925705 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:41.928189 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.928664 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.928667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.928668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.928745 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.929048 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.929051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.929053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.929277 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:41.929390 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.930133 (+ 743us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.930258 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4628,"thread_start_us":92,"threads_started":1}
I20260811 11:50:41.930509 16033 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:41.931137 16033 heartbeater.cc:511] Master 127.3.221.124:45801 requested a full tablet report, sending...
I20260811 11:50:41.931423 16034 heartbeater.cc:347] Connected to a master server at 127.3.221.126:46443
I20260811 11:50:41.931509 16034 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:41.931664 16034 heartbeater.cc:511] Master 127.3.221.126:46443 requested a full tablet report, sending...
I20260811 11:50:41.932289 14034 ts_manager.cc:194] Re-registered known tserver with Master: 777856fcd5864cffa63bd364422fc3a8 (127.3.221.67:44329)
I20260811 11:50:41.932327 15279 ts_manager.cc:194] Registered new tserver with Master: 777856fcd5864cffa63bd364422fc3a8 (127.3.221.67:44329)
I20260811 11:50:41.933049 14034 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:47231
I20260811 11:50:41.935933 16045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.911539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:39327 (local address 127.3.221.67:55843)
0811 11:50:41.912602 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.912613 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:41.913424 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:41.914218 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:41.914228 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:41.914893 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:41.915839 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.915860 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.916934 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.916937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.918094 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.918105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.918896 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.918899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.919068 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.919861 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:41.919874 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:41.922587 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:41.930656 (+ 8069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.930659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.930660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.930998 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.931337 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:41.931340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:41.931350 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:41.931449 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:41.931889 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:41.931892 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:41.935210 (+ 3318us) client_negotiation.cc:769] Sending connection context
0811 11:50:41.935574 (+ 364us) client_negotiation.cc:240] Negotiation successful
0811 11:50:41.935740 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"mutex_wait_us":482,"spinlock_wait_cycles":42624,"thread_start_us":130,"threads_started":1}
I20260811 11:50:41.936074 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.911357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:55843 (local address 127.3.221.125:39327)
0811 11:50:41.911595 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.911599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.912707 (+ 1108us) server_negotiation.cc:407] Connection header received
0811 11:50:41.913682 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.913685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.913741 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.913795 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:41.916003 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.916720 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.918215 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.918484 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.918594 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.923075 (+ 4481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:41.923079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:41.923082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:41.923108 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:41.926854 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.931111 (+ 4257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.931115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.931117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.931213 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.931544 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.931547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.931549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.931751 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:41.931827 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.935444 (+ 3617us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.935941 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260811 11:50:41.937206 16039 heartbeater.cc:347] Connected to a master server at 127.3.221.125:39327
I20260811 11:50:41.937352 16039 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:41.937597 16039 heartbeater.cc:511] Master 127.3.221.125:39327 requested a full tablet report, sending...
I20260811 11:50:41.938380 14983 ts_manager.cc:194] Registered new tserver with Master: 777856fcd5864cffa63bd364422fc3a8 (127.3.221.67:44329)
I20260811 11:50:41.939251 16052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.915173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47630 (local address 127.3.221.125:39327)
0811 11:50:41.918499 (+ 3326us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.918504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.918516 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:50:41.918553 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.918555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.918600 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.918680 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:41.920212 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.920945 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.931544 (+ 10599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.931898 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.931969 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.933952 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:41.933957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:41.933959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:41.933985 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:41.935834 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.937768 (+ 1934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.937772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.937773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.937882 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.938200 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.938203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.938205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.938435 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:41.938612 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.938981 (+ 369us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.939101 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3248,"thread_start_us":74,"threads_started":1}
I20260811 11:50:41.939896 16061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.916977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49988 (local address 127.3.221.126:46443)
0811 11:50:41.922068 (+ 5091us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.922072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.922086 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:50:41.922123 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.922126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.922170 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.922259 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:41.923386 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.924147 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.933956 (+ 9809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.934482 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.934569 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.936069 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:41.936073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:41.936076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:41.936102 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:41.937682 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.938155 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.938159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.938161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.938246 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:41.938502 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:41.938505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:41.938507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:41.938764 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:41.938900 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.939537 (+ 637us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.939731 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5006,"thread_start_us":81,"threads_started":1}
I20260811 11:50:41.942582 14034 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:50:41.945634 14034 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:50:41.948300 15824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b2b536c69864468947c78d7f24fea04" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: NON_VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: true } } }
I20260811 11:50:41.948464 15824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:41.951948 15824 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04: attempting to promote NON_VOTER 5ec79f33b453463abf6963e63ced5ac3 to VOTER
I20260811 11:50:41.952272 15824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: false } }
I20260811 11:50:41.953498 15305 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 1 LEARNER]: Refusing update from remote peer 9b2b536c69864468947c78d7f24fea04: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260811 11:50:41.953893 16065 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260811 11:50:41.955677 16065 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 5ec79f33b453463abf6963e63ced5ac3 (127.3.221.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: false } } }
I20260811 11:50:41.955974 15305 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 5ec79f33b453463abf6963e63ced5ac3 (127.3.221.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: false } } }
I20260811 11:50:41.956856 14997 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 1 FOLLOWER]: Refusing update from remote peer 9b2b536c69864468947c78d7f24fea04: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260811 11:50:41.957154 16066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b2b536c69864468947c78d7f24fea04" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: false } } }
I20260811 11:50:41.957331 16066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:41.957501 16063 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260811 11:50:41.957770 16065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9b2b536c69864468947c78d7f24fea04" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: false } } }
I20260811 11:50:41.957881 16065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:41.957870 15352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b2b536c69864468947c78d7f24fea04" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: false } } }
I20260811 11:50:41.957976 15352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:50:41.958648 14997 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 5ec79f33b453463abf6963e63ced5ac3 (127.3.221.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: false } } }
I20260811 11:50:41.960579 16065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b2b536c69864468947c78d7f24fea04" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: false } } }
I20260811 11:50:41.960695 16065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:41.964104 16079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b2b536c69864468947c78d7f24fea04" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: false } } }
I20260811 11:50:41.964253 16079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:50:41.966775 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.961634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:45063 (local address 127.0.0.1:33640)
0811 11:50:41.961753 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.961767 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:41.961851 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:41.962860 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:41.962863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:41.962880 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:41.962991 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.962999 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.964504 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.964508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.965296 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.965306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.966388 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.966392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.966475 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.966497 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 11:50:41.966558 (+ 61us) client_negotiation.cc:240] Negotiation successful
0811 11:50:41.966639 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260811 11:50:41.966795 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.961973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33640 (local address 127.3.221.65:45063)
0811 11:50:41.962284 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.962304 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.962332 (+ 28us) server_negotiation.cc:407] Connection header received
0811 11:50:41.962439 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.962445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.962694 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.963149 (+ 455us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:41.963297 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.964387 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.965453 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.966267 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.966366 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.966532 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.966586 (+ 54us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.966674 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":93,"threads_started":1}
I20260811 11:50:41.967799 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.962184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:40795 (local address 127.0.0.1:50596)
0811 11:50:41.962429 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.962444 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:41.962534 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:41.963648 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:41.963651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:41.963670 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:41.963776 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.963783 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.964883 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.964887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.965987 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.965996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.967445 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.967448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.967519 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.967536 (+ 17us) client_negotiation.cc:769] Sending connection context
0811 11:50:41.967593 (+ 57us) client_negotiation.cc:240] Negotiation successful
0811 11:50:41.967665 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":104,"threads_started":1}
I20260811 11:50:41.967887 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.962605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50596 (local address 127.3.221.66:40795)
0811 11:50:41.963026 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.963038 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.963061 (+ 23us) server_negotiation.cc:407] Connection header received
0811 11:50:41.963136 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.963142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.963334 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.963500 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:41.963921 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.964763 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.966119 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.966914 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.966990 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.967118 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.967652 (+ 534us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.967745 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":103,"threads_started":1}
I20260811 11:50:41.969913 16082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.962279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:44329 (local address 127.0.0.1:57528)
0811 11:50:41.963475 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:41.963489 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:41.963573 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:41.965587 (+ 2014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:41.965590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:41.965608 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:41.965712 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.965718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.967186 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.967189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.968327 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:41.968339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.969528 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.969531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:41.969622 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.969648 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 11:50:41.969696 (+ 48us) client_negotiation.cc:240] Negotiation successful
0811 11:50:41.969770 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1135,"thread_start_us":101,"threads_started":1}
I20260811 11:50:41.970556 16085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:41.962783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57528 (local address 127.3.221.67:44329)
0811 11:50:41.965083 (+ 2300us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:41.965102 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:41.965125 (+ 23us) server_negotiation.cc:407] Connection header received
0811 11:50:41.965200 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:41.965205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:41.965430 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:41.965868 (+ 438us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:41.966000 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.967069 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.968457 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:41.969385 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:41.970115 (+ 730us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:41.970267 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:41.970318 (+ 51us) server_negotiation.cc:299] Negotiation successful
0811 11:50:41.970408 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2190,"thread_start_us":96,"threads_started":1}
I20260811 11:50:41.970177 15475 tablet_service.cc:1467] Processing CreateTablet for tablet 4b8833b80941483b947706b1b134a333 (DEFAULT_TABLE table=test-table [id=2922818381a64be7bf36efc9d907030a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:41.970988 15700 tablet_service.cc:1467] Processing CreateTablet for tablet 4b8833b80941483b947706b1b134a333 (DEFAULT_TABLE table=test-table [id=2922818381a64be7bf36efc9d907030a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:41.971331 15475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b8833b80941483b947706b1b134a333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:41.971958 15700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b8833b80941483b947706b1b134a333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:41.973513 15922 tablet_service.cc:1467] Processing CreateTablet for tablet 4b8833b80941483b947706b1b134a333 (DEFAULT_TABLE table=test-table [id=2922818381a64be7bf36efc9d907030a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:41.974555 15922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b8833b80941483b947706b1b134a333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:41.979393 16086 tablet_bootstrap.cc:492] T 4b8833b80941483b947706b1b134a333 P 6065b7065aa84491a806b7276eb7b333: Bootstrap starting.
I20260811 11:50:41.981444 16086 tablet_bootstrap.cc:654] T 4b8833b80941483b947706b1b134a333 P 6065b7065aa84491a806b7276eb7b333: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:41.981627 16087 tablet_bootstrap.cc:492] T 4b8833b80941483b947706b1b134a333 P 09017061acfe48f6bb57b06e16ce084e: Bootstrap starting.
I20260811 11:50:41.982283 16086 log.cc:824] T 4b8833b80941483b947706b1b134a333 P 6065b7065aa84491a806b7276eb7b333: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:41.982362 16088 tablet_bootstrap.cc:492] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8: Bootstrap starting.
I20260811 11:50:41.983896 16087 tablet_bootstrap.cc:654] T 4b8833b80941483b947706b1b134a333 P 09017061acfe48f6bb57b06e16ce084e: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:41.984607 16086 tablet_bootstrap.cc:492] T 4b8833b80941483b947706b1b134a333 P 6065b7065aa84491a806b7276eb7b333: No bootstrap required, opened a new log
I20260811 11:50:41.984658 16088 tablet_bootstrap.cc:654] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:41.984793 16087 log.cc:824] T 4b8833b80941483b947706b1b134a333 P 09017061acfe48f6bb57b06e16ce084e: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:41.984826 16086 ts_tablet_manager.cc:1397] T 4b8833b80941483b947706b1b134a333 P 6065b7065aa84491a806b7276eb7b333: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 11:50:41.985469 16088 log.cc:824] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:41.986990 16087 tablet_bootstrap.cc:492] T 4b8833b80941483b947706b1b134a333 P 09017061acfe48f6bb57b06e16ce084e: No bootstrap required, opened a new log
I20260811 11:50:41.987144 16088 tablet_bootstrap.cc:492] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8: No bootstrap required, opened a new log
I20260811 11:50:41.987195 16087 ts_tablet_manager.cc:1397] T 4b8833b80941483b947706b1b134a333 P 09017061acfe48f6bb57b06e16ce084e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 11:50:41.987334 16088 ts_tablet_manager.cc:1397] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260811 11:50:41.989253 16086 raft_consensus.cc:348] T 4b8833b80941483b947706b1b134a333 P 6065b7065aa84491a806b7276eb7b333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6065b7065aa84491a806b7276eb7b333" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45063 } } peers { permanent_uuid: "777856fcd5864cffa63bd364422fc3a8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44329 } } peers { permanent_uuid: "09017061acfe48f6bb57b06e16ce084e" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40795 } }
I20260811 11:50:41.989552 16086 raft_consensus.cc:374] T 4b8833b80941483b947706b1b134a333 P 6065b7065aa84491a806b7276eb7b333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:41.989624 16086 raft_consensus.cc:729] T 4b8833b80941483b947706b1b134a333 P 6065b7065aa84491a806b7276eb7b333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6065b7065aa84491a806b7276eb7b333, State: Initialized, Role: FOLLOWER
I20260811 11:50:41.990140 16086 consensus_queue.cc:260] T 4b8833b80941483b947706b1b134a333 P 6065b7065aa84491a806b7276eb7b333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6065b7065aa84491a806b7276eb7b333" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45063 } } peers { permanent_uuid: "777856fcd5864cffa63bd364422fc3a8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44329 } } peers { permanent_uuid: "09017061acfe48f6bb57b06e16ce084e" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40795 } }
I20260811 11:50:41.991362 16086 ts_tablet_manager.cc:1428] T 4b8833b80941483b947706b1b134a333 P 6065b7065aa84491a806b7276eb7b333: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260811 11:50:41.991238 16088 raft_consensus.cc:348] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6065b7065aa84491a806b7276eb7b333" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45063 } } peers { permanent_uuid: "777856fcd5864cffa63bd364422fc3a8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44329 } } peers { permanent_uuid: "09017061acfe48f6bb57b06e16ce084e" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40795 } }
I20260811 11:50:41.991508 16088 raft_consensus.cc:374] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:41.991575 16088 raft_consensus.cc:729] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 777856fcd5864cffa63bd364422fc3a8, State: Initialized, Role: FOLLOWER
I20260811 11:50:41.991775 15593 heartbeater.cc:503] Master 127.3.221.126:46443 was elected leader, sending a full tablet report...
I20260811 11:50:41.992749 16088 consensus_queue.cc:260] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6065b7065aa84491a806b7276eb7b333" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45063 } } peers { permanent_uuid: "777856fcd5864cffa63bd364422fc3a8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44329 } } peers { permanent_uuid: "09017061acfe48f6bb57b06e16ce084e" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40795 } }
I20260811 11:50:41.993927 16088 ts_tablet_manager.cc:1428] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260811 11:50:41.994313 16034 heartbeater.cc:503] Master 127.3.221.126:46443 was elected leader, sending a full tablet report...
I20260811 11:50:41.992990 16087 raft_consensus.cc:348] T 4b8833b80941483b947706b1b134a333 P 09017061acfe48f6bb57b06e16ce084e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6065b7065aa84491a806b7276eb7b333" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45063 } } peers { permanent_uuid: "777856fcd5864cffa63bd364422fc3a8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44329 } } peers { permanent_uuid: "09017061acfe48f6bb57b06e16ce084e" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40795 } }
I20260811 11:50:41.996258 16087 raft_consensus.cc:374] T 4b8833b80941483b947706b1b134a333 P 09017061acfe48f6bb57b06e16ce084e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:41.996336 16087 raft_consensus.cc:729] T 4b8833b80941483b947706b1b134a333 P 09017061acfe48f6bb57b06e16ce084e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09017061acfe48f6bb57b06e16ce084e, State: Initialized, Role: FOLLOWER
I20260811 11:50:41.996835 16087 consensus_queue.cc:260] T 4b8833b80941483b947706b1b134a333 P 09017061acfe48f6bb57b06e16ce084e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6065b7065aa84491a806b7276eb7b333" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45063 } } peers { permanent_uuid: "777856fcd5864cffa63bd364422fc3a8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44329 } } peers { permanent_uuid: "09017061acfe48f6bb57b06e16ce084e" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40795 } }
I20260811 11:50:41.997936 16087 ts_tablet_manager.cc:1428] T 4b8833b80941483b947706b1b134a333 P 09017061acfe48f6bb57b06e16ce084e: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260811 11:50:41.998410 15814 heartbeater.cc:503] Master 127.3.221.126:46443 was elected leader, sending a full tablet report...
I20260811 11:50:42.143309 16098 raft_consensus.cc:482] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:50:42.143589 16098 raft_consensus.cc:504] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6065b7065aa84491a806b7276eb7b333" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45063 } } peers { permanent_uuid: "777856fcd5864cffa63bd364422fc3a8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44329 } } peers { permanent_uuid: "09017061acfe48f6bb57b06e16ce084e" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40795 } }
I20260811 11:50:42.144783 16098 leader_election.cc:288] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6065b7065aa84491a806b7276eb7b333 (127.3.221.65:45063), 09017061acfe48f6bb57b06e16ce084e (127.3.221.66:40795)
I20260811 11:50:42.148736 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:42.145475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:40111 (local address 127.3.221.66:40795)
0811 11:50:42.145610 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:42.145615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:42.145629 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:50:42.145677 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:42.145680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:42.145721 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:42.145799 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:42.146110 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:42.146670 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:42.147785 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:42.148170 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:42.148266 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:42.148321 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:42.148525 (+ 204us) server_negotiation.cc:299] Negotiation successful
0811 11:50:42.148602 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 11:50:42.148738 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:42.145183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:48481 (local address 127.3.221.65:45063)
0811 11:50:42.145356 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:42.145361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:42.145380 (+ 19us) server_negotiation.cc:407] Connection header received
0811 11:50:42.145493 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:42.145496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:42.145552 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:42.145622 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:42.145958 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:42.146591 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:42.147820 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:42.148222 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:42.148351 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:42.148403 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:42.148557 (+ 154us) server_negotiation.cc:299] Negotiation successful
0811 11:50:42.148628 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260811 11:50:42.148738 16045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:42.145104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:45063 (local address 127.3.221.67:48481)
0811 11:50:42.145308 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:42.145320 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:42.145388 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:42.145657 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:42.145660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:42.145680 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:42.145835 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:42.145854 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:42.146699 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:42.146702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:42.147687 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:42.147697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:42.148349 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:42.148353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:42.148441 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:42.148458 (+ 17us) client_negotiation.cc:769] Sending connection context
0811 11:50:42.148532 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 11:50:42.148602 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":41,"spinlock_wait_cycles":1792}
I20260811 11:50:42.148761 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:42.145075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:40795 (local address 127.3.221.67:40111)
0811 11:50:42.145256 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:42.145273 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:42.145381 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:42.145830 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:42.145844 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:42.145864 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:42.146003 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:42.146010 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:42.146769 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:42.146771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:42.147649 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:42.147660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:42.148289 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:42.148294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:42.148398 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:42.148422 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 11:50:42.148488 (+ 66us) client_negotiation.cc:240] Negotiation successful
0811 11:50:42.148562 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":2}
I20260811 11:50:42.149487 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b8833b80941483b947706b1b134a333" candidate_uuid: "777856fcd5864cffa63bd364422fc3a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09017061acfe48f6bb57b06e16ce084e" is_pre_election: true
I20260811 11:50:42.149484 15521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b8833b80941483b947706b1b134a333" candidate_uuid: "777856fcd5864cffa63bd364422fc3a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6065b7065aa84491a806b7276eb7b333" is_pre_election: true
I20260811 11:50:42.149844 15737 raft_consensus.cc:2454] T 4b8833b80941483b947706b1b134a333 P 09017061acfe48f6bb57b06e16ce084e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 777856fcd5864cffa63bd364422fc3a8 in term 0.
I20260811 11:50:42.149834 15521 raft_consensus.cc:2454] T 4b8833b80941483b947706b1b134a333 P 6065b7065aa84491a806b7276eb7b333 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 777856fcd5864cffa63bd364422fc3a8 in term 0.
I20260811 11:50:42.150378 15848 leader_election.cc:302] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09017061acfe48f6bb57b06e16ce084e, 777856fcd5864cffa63bd364422fc3a8; no voters:
I20260811 11:50:42.150668 16098 raft_consensus.cc:2781] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:50:42.150758 16098 raft_consensus.cc:482] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:50:42.150808 16098 raft_consensus.cc:3037] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:42.151943 16098 raft_consensus.cc:504] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6065b7065aa84491a806b7276eb7b333" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45063 } } peers { permanent_uuid: "777856fcd5864cffa63bd364422fc3a8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44329 } } peers { permanent_uuid: "09017061acfe48f6bb57b06e16ce084e" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40795 } }
I20260811 11:50:42.152283 16098 leader_election.cc:288] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8 [CANDIDATE]: Term 1 election: Requested vote from peers 6065b7065aa84491a806b7276eb7b333 (127.3.221.65:45063), 09017061acfe48f6bb57b06e16ce084e (127.3.221.66:40795)
I20260811 11:50:42.152642 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b8833b80941483b947706b1b134a333" candidate_uuid: "777856fcd5864cffa63bd364422fc3a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09017061acfe48f6bb57b06e16ce084e"
I20260811 11:50:42.152653 15521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b8833b80941483b947706b1b134a333" candidate_uuid: "777856fcd5864cffa63bd364422fc3a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6065b7065aa84491a806b7276eb7b333"
I20260811 11:50:42.152762 15737 raft_consensus.cc:3037] T 4b8833b80941483b947706b1b134a333 P 09017061acfe48f6bb57b06e16ce084e [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:42.152786 15521 raft_consensus.cc:3037] T 4b8833b80941483b947706b1b134a333 P 6065b7065aa84491a806b7276eb7b333 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:42.153892 15737 raft_consensus.cc:2454] T 4b8833b80941483b947706b1b134a333 P 09017061acfe48f6bb57b06e16ce084e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 777856fcd5864cffa63bd364422fc3a8 in term 1.
I20260811 11:50:42.153947 15521 raft_consensus.cc:2454] T 4b8833b80941483b947706b1b134a333 P 6065b7065aa84491a806b7276eb7b333 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 777856fcd5864cffa63bd364422fc3a8 in term 1.
I20260811 11:50:42.154158 15848 leader_election.cc:302] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09017061acfe48f6bb57b06e16ce084e, 777856fcd5864cffa63bd364422fc3a8; no voters:
I20260811 11:50:42.154330 16098 raft_consensus.cc:2781] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:42.154567 16098 raft_consensus.cc:686] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8 [term 1 LEADER]: Becoming Leader. State: Replica: 777856fcd5864cffa63bd364422fc3a8, State: Running, Role: LEADER
I20260811 11:50:42.154984 16098 consensus_queue.cc:237] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6065b7065aa84491a806b7276eb7b333" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45063 } } peers { permanent_uuid: "777856fcd5864cffa63bd364422fc3a8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44329 } } peers { permanent_uuid: "09017061acfe48f6bb57b06e16ce084e" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40795 } }
I20260811 11:50:42.158397 14033 catalog_manager.cc:5495] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 777856fcd5864cffa63bd364422fc3a8 (127.3.221.67). New cstate: current_term: 1 leader_uuid: "777856fcd5864cffa63bd364422fc3a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6065b7065aa84491a806b7276eb7b333" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "777856fcd5864cffa63bd364422fc3a8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09017061acfe48f6bb57b06e16ce084e" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40795 } health_report { overall_health: UNKNOWN } } }
W20260811 11:50:42.160902 16041 tablet.cc:2369] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:50:42.184720 15817 tablet.cc:2369] T 4b8833b80941483b947706b1b134a333 P 09017061acfe48f6bb57b06e16ce084e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:50:42.203557 15595 tablet.cc:2369] T 4b8833b80941483b947706b1b134a333 P 6065b7065aa84491a806b7276eb7b333: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:50:42.204756 16085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:42.201149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57530 (local address 127.3.221.67:44329)
0811 11:50:42.201316 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:42.201319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:42.201332 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:50:42.201417 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:42.201419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:42.201476 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:42.201593 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:42.201826 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:42.202336 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:42.203189 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:42.203403 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:42.203493 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:42.203856 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:42.204400 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:42.204502 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:42.204585 (+ 83us) server_negotiation.cc:299] Negotiation successful
0811 11:50:42.204642 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 11:50:42.210906 15737 raft_consensus.cc:1261] T 4b8833b80941483b947706b1b134a333 P 09017061acfe48f6bb57b06e16ce084e [term 1 FOLLOWER]: Refusing update from remote peer 777856fcd5864cffa63bd364422fc3a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:42.210906 15521 raft_consensus.cc:1261] T 4b8833b80941483b947706b1b134a333 P 6065b7065aa84491a806b7276eb7b333 [term 1 FOLLOWER]: Refusing update from remote peer 777856fcd5864cffa63bd364422fc3a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:42.211773 16098 consensus_queue.cc:1035] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6065b7065aa84491a806b7276eb7b333" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:42.211982 16100 consensus_queue.cc:1035] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09017061acfe48f6bb57b06e16ce084e" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:42.222223 16108 mvcc.cc:205] Tried to move back new op lower bound from 7317295276887670784 to 7317295276669681664. Current Snapshot: MvccSnapshot[applied={T|T < 7317295276887670784}]
I20260811 11:50:42.222440 16107 mvcc.cc:205] Tried to move back new op lower bound from 7317295276887670784 to 7317295276669681664. Current Snapshot: MvccSnapshot[applied={T|T < 7317295276887670784}]
I20260811 11:50:42.233388 14033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:42.233595 14033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:42.238847 14033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b2b536c69864468947c78d7f24fea04: Sending DeleteTablet for 3 replicas of tablet 4b8833b80941483b947706b1b134a333
I20260811 11:50:42.239674 15922 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b8833b80941483b947706b1b134a333 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:42 UTC) from {username='oryx', principal='oryx/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:57528
I20260811 11:50:42.239771 15475 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b8833b80941483b947706b1b134a333 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:42 UTC) from {username='oryx', principal='oryx/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:33640
I20260811 11:50:42.239924 15700 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b8833b80941483b947706b1b134a333 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:42 UTC) from {username='oryx', principal='oryx/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:50596
I20260811 11:50:42.240747 16125 tablet_replica.cc:330] T 4b8833b80941483b947706b1b134a333 P 09017061acfe48f6bb57b06e16ce084e: stopping tablet replica
I20260811 11:50:42.241014 16123 tablet_replica.cc:330] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8: stopping tablet replica
I20260811 11:50:42.241369 16123 raft_consensus.cc:2229] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:50:42.241743 16125 raft_consensus.cc:2229] T 4b8833b80941483b947706b1b134a333 P 09017061acfe48f6bb57b06e16ce084e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:42.241979 16123 raft_consensus.cc:2258] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:42.242682 16125 raft_consensus.cc:2258] T 4b8833b80941483b947706b1b134a333 P 09017061acfe48f6bb57b06e16ce084e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:42.242719 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 13985
I20260811 11:50:42.256223 16124 tablet_replica.cc:330] T 4b8833b80941483b947706b1b134a333 P 6065b7065aa84491a806b7276eb7b333: stopping tablet replica
I20260811 11:50:42.256656 16124 raft_consensus.cc:2229] T 4b8833b80941483b947706b1b134a333 P 6065b7065aa84491a806b7276eb7b333 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:42.257064 16124 raft_consensus.cc:2258] T 4b8833b80941483b947706b1b134a333 P 6065b7065aa84491a806b7276eb7b333 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:42.258566 16125 ts_tablet_manager.cc:1905] T 4b8833b80941483b947706b1b134a333 P 09017061acfe48f6bb57b06e16ce084e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:42.261564 16125 ts_tablet_manager.cc:1918] T 4b8833b80941483b947706b1b134a333 P 09017061acfe48f6bb57b06e16ce084e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:50:42.261688 16125 log.cc:1196] T 4b8833b80941483b947706b1b134a333 P 09017061acfe48f6bb57b06e16ce084e: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/4b8833b80941483b947706b1b134a333
I20260811 11:50:42.262059 16125 ts_tablet_manager.cc:1939] T 4b8833b80941483b947706b1b134a333 P 09017061acfe48f6bb57b06e16ce084e: Deleting consensus metadata
I20260811 11:50:42.262933 16123 ts_tablet_manager.cc:1905] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:42.265024 16124 ts_tablet_manager.cc:1905] T 4b8833b80941483b947706b1b134a333 P 6065b7065aa84491a806b7276eb7b333: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:42.266067 16123 ts_tablet_manager.cc:1918] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:50:42.266248 16123 log.cc:1196] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/4b8833b80941483b947706b1b134a333
I20260811 11:50:42.266633 16123 ts_tablet_manager.cc:1939] T 4b8833b80941483b947706b1b134a333 P 777856fcd5864cffa63bd364422fc3a8: Deleting consensus metadata
I20260811 11:50:42.267705 16124 ts_tablet_manager.cc:1918] T 4b8833b80941483b947706b1b134a333 P 6065b7065aa84491a806b7276eb7b333: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:50:42.267812 16124 log.cc:1196] T 4b8833b80941483b947706b1b134a333 P 6065b7065aa84491a806b7276eb7b333: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-0/wal/wals/4b8833b80941483b947706b1b134a333
I20260811 11:50:42.268123 16124 ts_tablet_manager.cc:1939] T 4b8833b80941483b947706b1b134a333 P 6065b7065aa84491a806b7276eb7b333: Deleting consensus metadata
W20260811 11:50:42.268289 15630 connection.cc:535] server connection from 127.0.0.1:50596 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260811 11:50:42.268575 15851 connection.cc:441] server connection from 127.0.0.1:57528 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57528 (request call id 1) could send its response
W20260811 11:50:42.270032 15405 connection.cc:441] server connection from 127.0.0.1:33640 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33640 (request call id 1) could send its response
I20260811 11:50:42.282271 16052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:42.241321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47636 (local address 127.3.221.125:39327)
0811 11:50:42.241447 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:42.241452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:42.268894 (+ 27442us) server_negotiation.cc:407] Connection header received
0811 11:50:42.269128 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:42.269132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:42.269195 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:42.269270 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:42.270786 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:42.271586 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:42.272763 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:42.273099 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:42.273212 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:42.275126 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:42.275130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:42.275133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:42.275161 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:42.276906 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:42.277442 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:42.277445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:42.277447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:42.277540 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:42.277987 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:42.277991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:42.277993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:42.278240 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:42.278360 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:42.279851 (+ 1491us) server_negotiation.cc:299] Negotiation successful
0811 11:50:42.279991 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53,"spinlock_wait_cycles":5376}
I20260811 11:50:42.284721 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:42.272877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38790 (local address 127.3.221.124:45801)
0811 11:50:42.272999 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:42.273003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:42.273742 (+ 739us) server_negotiation.cc:407] Connection header received
0811 11:50:42.273996 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:42.274000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:42.274050 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:42.274101 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:42.275080 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:42.275882 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:42.277749 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:42.278061 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:42.278138 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:42.279986 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:42.279990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:42.279992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:42.280017 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:42.282305 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:42.283299 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:42.283303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:42.283305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:42.283389 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:42.283706 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:42.283709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:42.283711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:42.283935 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:42.284045 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:42.284425 (+ 380us) server_negotiation.cc:299] Negotiation successful
0811 11:50:42.284553 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 11:50:42.285197 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:42.272905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47638 (local address 127.3.221.125:39327)
0811 11:50:42.273022 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:42.273026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:42.273452 (+ 426us) server_negotiation.cc:407] Connection header received
0811 11:50:42.273573 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:42.273577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:42.273627 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:42.273701 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:42.275228 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:42.275955 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:42.277704 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:42.278049 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:42.278149 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:42.280027 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:42.280031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:42.280034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:42.280061 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:42.282074 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:42.283691 (+ 1617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:42.283695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:42.283697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:42.283867 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:42.284258 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:42.284261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:42.284263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:42.284484 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:42.284583 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:42.284940 (+ 357us) server_negotiation.cc:299] Negotiation successful
0811 11:50:42.285053 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 11:50:42.287405 16054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:42.241635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38784 (local address 127.3.221.124:45801)
0811 11:50:42.241754 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:42.241759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:42.279130 (+ 37371us) server_negotiation.cc:407] Connection header received
0811 11:50:42.279304 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:42.279308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:42.279365 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:42.279417 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:42.280519 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:42.281023 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:42.282168 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:42.282514 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:42.282577 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:42.284118 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:42.284122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:42.284124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:42.284148 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:42.285719 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:42.286177 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:42.286180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:42.286182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:42.286268 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:42.286593 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:42.286596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:42.286598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:42.286809 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:42.286935 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:42.287123 (+ 188us) server_negotiation.cc:299] Negotiation successful
0811 11:50:42.287246 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 11:50:43.225459 16143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:43.224787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.66:52887)
0811 11:50:43.225169 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:43.225328 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":88,"threads_started":1}
W20260811 11:50:43.225898 15814 heartbeater.cc:644] Failed to heartbeat to 127.3.221.126:46443 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
I20260811 11:50:43.226969 16146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:43.226479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.65:43099)
0811 11:50:43.226763 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:43.226857 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":71,"threads_started":1}
W20260811 11:50:43.227257 15593 heartbeater.cc:644] Failed to heartbeat to 127.3.221.126:46443 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
I20260811 11:50:43.229018 16149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:43.228634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.67:38197)
0811 11:50:43.228855 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:43.228928 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":79,"threads_started":1}
W20260811 11:50:43.229348 16034 heartbeater.cc:644] Failed to heartbeat to 127.3.221.126:46443 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
I20260811 11:50:43.738618 16150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b2b536c69864468947c78d7f24fea04)
I20260811 11:50:43.738854 16150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: false } }
I20260811 11:50:43.739130 16151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b2b536c69864468947c78d7f24fea04)
I20260811 11:50:43.739295 16151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: false } }
I20260811 11:50:43.740027 16150 leader_election.cc:288] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b2b536c69864468947c78d7f24fea04 (127.3.221.126:46443), 30ce91da8cce49dba59fb1865beb961c (127.3.221.125:39327)
I20260811 11:50:43.740278 16151 leader_election.cc:288] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b2b536c69864468947c78d7f24fea04 (127.3.221.126:46443), 5ec79f33b453463abf6963e63ced5ac3 (127.3.221.124:45801)
I20260811 11:50:43.740780 16153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:43.740164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.0.0.1:50506)
0811 11:50:43.740531 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:43.740618 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":79,"threads_started":1}
W20260811 11:50:43.740950 15268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
I20260811 11:50:43.741187 16159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:43.740537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.0.0.1:50518)
0811 11:50:43.741029 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:43.741100 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":45,"threads_started":1}
W20260811 11:50:43.741361 14963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
I20260811 11:50:43.741984 16153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:43.741716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.0.0.1:50528)
0811 11:50:43.741854 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:43.741916 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260811 11:50:43.742128 15268 leader_election.cc:334] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9b2b536c69864468947c78d7f24fea04 (127.3.221.126:46443): Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
I20260811 11:50:43.742292 16159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:43.742043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.0.0.1:50540)
0811 11:50:43.742179 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:43.742228 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260811 11:50:43.742436 14963 leader_election.cc:334] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9b2b536c69864468947c78d7f24fea04 (127.3.221.126:46443): Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
I20260811 11:50:43.745442 16158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:43.740480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:45801 (local address 127.0.0.1:38796)
0811 11:50:43.740739 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:43.740751 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:43.740856 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:43.741872 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:43.741875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:43.741891 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:43.741971 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:43.741976 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:43.742993 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:43.742997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:43.744200 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:43.744210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:43.744990 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:43.744994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:43.745078 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:43.745104 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 11:50:43.745176 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 11:50:43.745273 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":3,"thread_start_us":58,"threads_started":1}
I20260811 11:50:43.745450 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:43.740226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:39327 (local address 127.0.0.1:47644)
0811 11:50:43.741021 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:43.741035 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:43.741136 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:43.741622 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:43.741625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:43.741645 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:43.741749 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:43.741756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:43.742790 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:43.742793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:43.744033 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:43.744045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:43.744940 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:43.744944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:43.745078 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:43.745104 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 11:50:43.745180 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 11:50:43.745281 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":21,"thread_start_us":78,"threads_started":1}
I20260811 11:50:43.745452 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:43.740857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38796 (local address 127.3.221.124:45801)
0811 11:50:43.741595 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:43.741599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:43.741609 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:50:43.741641 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:43.741644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:43.741689 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:43.741783 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:43.742183 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:43.742896 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:43.744305 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:43.744891 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:43.744992 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:43.745077 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:43.745207 (+ 130us) server_negotiation.cc:299] Negotiation successful
0811 11:50:43.745303 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":670,"spinlock_wait_cycles":5376,"thread_start_us":79,"threads_started":1}
I20260811 11:50:43.745451 16156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:43.740430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47644 (local address 127.3.221.125:39327)
0811 11:50:43.740680 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:43.740717 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:43.741254 (+ 537us) server_negotiation.cc:407] Connection header received
0811 11:50:43.741310 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:43.741313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:43.741361 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:43.741491 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:43.741867 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:43.742664 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:43.744165 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:43.744811 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:43.744919 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:43.744986 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:43.745208 (+ 222us) server_negotiation.cc:299] Negotiation successful
0811 11:50:43.745303 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":182,"thread_start_us":68,"threads_started":1}
I20260811 11:50:43.746121 15305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30ce91da8cce49dba59fb1865beb961c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5ec79f33b453463abf6963e63ced5ac3" is_pre_election: true
I20260811 11:50:43.746245 14997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5ec79f33b453463abf6963e63ced5ac3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "30ce91da8cce49dba59fb1865beb961c" is_pre_election: true
I20260811 11:50:43.746345 15305 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ce91da8cce49dba59fb1865beb961c in term 1.
I20260811 11:50:43.746412 14997 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ec79f33b453463abf6963e63ced5ac3 in term 1.
I20260811 11:50:43.746711 14961 leader_election.cc:302] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ce91da8cce49dba59fb1865beb961c, 5ec79f33b453463abf6963e63ced5ac3; no voters: 9b2b536c69864468947c78d7f24fea04
I20260811 11:50:43.746729 15262 leader_election.cc:302] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ce91da8cce49dba59fb1865beb961c, 5ec79f33b453463abf6963e63ced5ac3; no voters: 9b2b536c69864468947c78d7f24fea04
I20260811 11:50:43.746940 16151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260811 11:50:43.746951 16150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260811 11:50:43.747004 16150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9b2b536c69864468947c78d7f24fea04)
I20260811 11:50:43.747004 16151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9b2b536c69864468947c78d7f24fea04)
I20260811 11:50:43.747045 16151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 1 FOLLOWER]: Advancing to term 2
I20260811 11:50:43.747045 16150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 1 FOLLOWER]: Advancing to term 2
I20260811 11:50:43.748309 16150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: false } }
I20260811 11:50:43.748309 16151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: false } }
I20260811 11:50:43.748703 16150 leader_election.cc:288] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [CANDIDATE]: Term 2 election: Requested vote from peers 9b2b536c69864468947c78d7f24fea04 (127.3.221.126:46443), 30ce91da8cce49dba59fb1865beb961c (127.3.221.125:39327)
I20260811 11:50:43.748703 16151 leader_election.cc:288] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [CANDIDATE]: Term 2 election: Requested vote from peers 9b2b536c69864468947c78d7f24fea04 (127.3.221.126:46443), 5ec79f33b453463abf6963e63ced5ac3 (127.3.221.124:45801)
I20260811 11:50:43.749089 16158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:43.748852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.0.0.1:50556)
0811 11:50:43.748960 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:43.749018 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260811 11:50:43.749248 15305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30ce91da8cce49dba59fb1865beb961c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5ec79f33b453463abf6963e63ced5ac3"
I20260811 11:50:43.749282 14997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5ec79f33b453463abf6963e63ced5ac3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "30ce91da8cce49dba59fb1865beb961c"
I20260811 11:50:43.749425 15305 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 30ce91da8cce49dba59fb1865beb961c in current term 2: Already voted for candidate 5ec79f33b453463abf6963e63ced5ac3 in this term.
I20260811 11:50:43.749459 14997 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ec79f33b453463abf6963e63ced5ac3 in current term 2: Already voted for candidate 30ce91da8cce49dba59fb1865beb961c in this term.
I20260811 11:50:43.749702 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:43.748846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.0.0.1:50554)
0811 11:50:43.749162 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:43.749634 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249}
I20260811 11:50:43.750156 16158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:43.749950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.0.0.1:50574)
0811 11:50:43.750036 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:43.750091 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260811 11:50:43.750250 14963 leader_election.cc:334] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9b2b536c69864468947c78d7f24fea04 (127.3.221.126:46443): Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
I20260811 11:50:43.750310 14963 leader_election.cc:302] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ce91da8cce49dba59fb1865beb961c; no voters: 5ec79f33b453463abf6963e63ced5ac3, 9b2b536c69864468947c78d7f24fea04
I20260811 11:50:43.750422 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:43.750181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.0.0.1:50580)
0811 11:50:43.750288 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:43.750349 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260811 11:50:43.750439 16151 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260811 11:50:43.750500 15268 leader_election.cc:334] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9b2b536c69864468947c78d7f24fea04 (127.3.221.126:46443): Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
I20260811 11:50:43.750559 15268 leader_election.cc:302] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ec79f33b453463abf6963e63ced5ac3; no voters: 30ce91da8cce49dba59fb1865beb961c, 9b2b536c69864468947c78d7f24fea04
I20260811 11:50:43.750736 16150 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260811 11:50:44.227070 16168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:44.226554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.66:51381)
0811 11:50:44.226875 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:44.226955 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":116,"threads_started":1}
I20260811 11:50:44.228170 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:44.227771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.65:52895)
0811 11:50:44.228031 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:44.228085 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":90,"threads_started":1}
I20260811 11:50:44.230180 16170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:44.229831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.67:41679)
0811 11:50:44.230043 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:44.230098 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":66,"threads_started":1}
I20260811 11:50:45.228338 16171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.227924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.66:42957)
0811 11:50:45.228177 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:45.228239 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
I20260811 11:50:45.229362 16172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.228926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.65:45327)
0811 11:50:45.229159 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:45.229262 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":66,"threads_started":1}
I20260811 11:50:45.231112 16173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.230747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.67:56929)
0811 11:50:45.230973 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:45.231029 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":65,"threads_started":1}
I20260811 11:50:45.562884 16174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:50:45.563040 16174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: false } }
I20260811 11:50:45.563508 16174 leader_election.cc:288] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9b2b536c69864468947c78d7f24fea04 (127.3.221.126:46443), 5ec79f33b453463abf6963e63ced5ac3 (127.3.221.124:45801)
I20260811 11:50:45.564002 15305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30ce91da8cce49dba59fb1865beb961c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5ec79f33b453463abf6963e63ced5ac3" is_pre_election: true
I20260811 11:50:45.564095 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.563720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.0.0.1:50868)
0811 11:50:45.563940 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:45.564001 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":60,"threads_started":1}
I20260811 11:50:45.564157 15305 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ce91da8cce49dba59fb1865beb961c in term 2.
I20260811 11:50:45.564512 14961 leader_election.cc:302] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ce91da8cce49dba59fb1865beb961c, 5ec79f33b453463abf6963e63ced5ac3; no voters:
I20260811 11:50:45.564692 16174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260811 11:50:45.564750 16174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:50:45.564790 16174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 2 FOLLOWER]: Advancing to term 3
I20260811 11:50:45.565019 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.564807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.0.0.1:50882)
0811 11:50:45.564914 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:45.564960 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260811 11:50:45.565120 14963 leader_election.cc:334] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b2b536c69864468947c78d7f24fea04 (127.3.221.126:46443): Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
I20260811 11:50:45.565955 16174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: false } }
I20260811 11:50:45.566278 16174 leader_election.cc:288] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [CANDIDATE]: Term 3 election: Requested vote from peers 9b2b536c69864468947c78d7f24fea04 (127.3.221.126:46443), 5ec79f33b453463abf6963e63ced5ac3 (127.3.221.124:45801)
I20260811 11:50:45.566624 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.566392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.0.0.1:50898)
0811 11:50:45.566505 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:45.566560 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260811 11:50:45.566663 15305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30ce91da8cce49dba59fb1865beb961c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5ec79f33b453463abf6963e63ced5ac3"
I20260811 11:50:45.566802 15305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 2 FOLLOWER]: Advancing to term 3
I20260811 11:50:45.567315 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.567102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.0.0.1:50904)
0811 11:50:45.567215 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:45.567265 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260811 11:50:45.567418 14963 leader_election.cc:334] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9b2b536c69864468947c78d7f24fea04 (127.3.221.126:46443): Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
I20260811 11:50:45.567961 15305 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ce91da8cce49dba59fb1865beb961c in term 3.
I20260811 11:50:45.568217 14961 leader_election.cc:302] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ce91da8cce49dba59fb1865beb961c, 5ec79f33b453463abf6963e63ced5ac3; no voters: 9b2b536c69864468947c78d7f24fea04
I20260811 11:50:45.568404 16174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 3 FOLLOWER]: Leader election won for term 3
I20260811 11:50:45.568585 16174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [term 3 LEADER]: Becoming Leader. State: Replica: 30ce91da8cce49dba59fb1865beb961c, State: Running, Role: LEADER
I20260811 11:50:45.568842 16174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: false } }
I20260811 11:50:45.569975 16180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30ce91da8cce49dba59fb1865beb961c. Latest consensus state: current_term: 3 leader_uuid: "30ce91da8cce49dba59fb1865beb961c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: false } } }
I20260811 11:50:45.570111 16180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:45.570399 16182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:50:45.573504 16182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:50:45.573959 16182 catalog_manager.cc:1259] Loaded cluster ID: 93c1574656b04143adc85e20ebe4d8ad
I20260811 11:50:45.574007 16182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:50:45.574559 16182 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:50:45.574951 16182 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c: Loaded TSK: 0
I20260811 11:50:45.575641 16182 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:50:45.657557 14982 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:50:45.660569 14982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:50:45.663903 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.663652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.0.0.1:50932)
0811 11:50:45.663769 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:45.663829 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260811 11:50:45.663959 15305 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [term 3 FOLLOWER]: Refusing update from remote peer 30ce91da8cce49dba59fb1865beb961c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260811 11:50:45.664471 16180 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260811 11:50:45.664601 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.664391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.0.0.1:50944)
0811 11:50:45.664495 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:45.664544 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260811 11:50:45.664798 14963 consensus_peers.cc:487] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c -> Peer 9b2b536c69864468947c78d7f24fea04 (127.3.221.126:46443): Couldn't send request to peer 9b2b536c69864468947c78d7f24fea04. Status: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260811 11:50:45.666033 16184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30ce91da8cce49dba59fb1865beb961c. Latest consensus state: current_term: 3 leader_uuid: "30ce91da8cce49dba59fb1865beb961c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: false } } }
I20260811 11:50:45.666163 16184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:50:45.667157 16180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "30ce91da8cce49dba59fb1865beb961c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: false } } }
I20260811 11:50:45.667285 16180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:45.667886 16184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "30ce91da8cce49dba59fb1865beb961c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b2b536c69864468947c78d7f24fea04" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46443 } } peers { permanent_uuid: "30ce91da8cce49dba59fb1865beb961c" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "5ec79f33b453463abf6963e63ced5ac3" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 45801 } attrs { promote: false } } }
I20260811 11:50:45.668058 16184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec79f33b453463abf6963e63ced5ac3 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:50:45.677834 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.673072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:45063 (local address 127.0.0.1:33648)
0811 11:50:45.673188 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:45.673203 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:45.673315 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:45.674373 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:45.674376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:45.674391 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:45.674483 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:45.674488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:45.675445 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:45.675449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:45.676533 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:45.676542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:45.677450 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:45.677454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:45.677542 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:45.677570 (+ 28us) client_negotiation.cc:769] Sending connection context
0811 11:50:45.677617 (+ 47us) client_negotiation.cc:240] Negotiation successful
0811 11:50:45.677685 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260811 11:50:45.678622 16196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.673642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:40795 (local address 127.0.0.1:50604)
0811 11:50:45.674151 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:45.674165 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:45.674286 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:45.674800 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:45.674802 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:45.674816 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:45.674883 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:45.674887 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:45.675712 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:45.675715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:45.677144 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:45.677153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:45.678143 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:45.678147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:45.678287 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:45.678338 (+ 51us) client_negotiation.cc:769] Sending connection context
0811 11:50:45.678403 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 11:50:45.678486 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":32,"spinlock_wait_cycles":2688,"thread_start_us":128,"threads_started":1}
I20260811 11:50:45.678622 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.673366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33648 (local address 127.3.221.65:45063)
0811 11:50:45.674112 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:45.674117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:45.674132 (+ 15us) server_negotiation.cc:407] Connection header received
0811 11:50:45.674185 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:45.674188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:45.674238 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:45.674356 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:45.674593 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:45.675338 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:45.676664 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:45.677325 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:45.678336 (+ 1011us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:45.678400 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:45.678435 (+ 35us) server_negotiation.cc:299] Negotiation successful
0811 11:50:45.678505 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":78,"threads_started":1}
I20260811 11:50:45.678676 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.674051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57534 (local address 127.3.221.67:44329)
0811 11:50:45.675003 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:45.675007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:45.675020 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:50:45.675068 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:45.675071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:45.675112 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:45.675223 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:45.675492 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:45.676200 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:45.677534 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:45.678025 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:45.678121 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:45.678188 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:45.678486 (+ 298us) server_negotiation.cc:299] Negotiation successful
0811 11:50:45.678562 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":876,"thread_start_us":71,"threads_started":1}
I20260811 11:50:45.678674 16200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.673880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50604 (local address 127.3.221.66:40795)
0811 11:50:45.674560 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:45.674564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:45.674577 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:50:45.674623 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:45.674626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:45.674669 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:45.674785 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:45.674994 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:45.675607 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:45.677302 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:45.677913 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:45.677994 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:45.678061 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:45.678441 (+ 380us) server_negotiation.cc:299] Negotiation successful
0811 11:50:45.678523 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":81,"threads_started":1}
I20260811 11:50:45.679014 16197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.673551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:44329 (local address 127.0.0.1:57534)
0811 11:50:45.674131 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:45.674145 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:45.674281 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:45.675246 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:45.675250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:45.675268 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:45.675361 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:45.675367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:45.676317 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:45.676320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:45.677423 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:45.677433 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:45.678142 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:45.678146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:45.678302 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:45.678353 (+ 51us) client_negotiation.cc:769] Sending connection context
0811 11:50:45.678418 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 11:50:45.678878 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":409,"spinlock_wait_cycles":11904,"thread_start_us":97,"threads_started":1}
I20260811 11:50:45.679594 15475 tablet_service.cc:1467] Processing CreateTablet for tablet 03eaa215fb63484dad09405b9867ae4d (DEFAULT_TABLE table=test-table [id=b7e7b690770f44d1aff901b235d06950]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:45.679594 15922 tablet_service.cc:1467] Processing CreateTablet for tablet 03eaa215fb63484dad09405b9867ae4d (DEFAULT_TABLE table=test-table [id=b7e7b690770f44d1aff901b235d06950]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:45.679893 15922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03eaa215fb63484dad09405b9867ae4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:45.679893 15475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03eaa215fb63484dad09405b9867ae4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:45.679848 15700 tablet_service.cc:1467] Processing CreateTablet for tablet 03eaa215fb63484dad09405b9867ae4d (DEFAULT_TABLE table=test-table [id=b7e7b690770f44d1aff901b235d06950]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:45.680075 15700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03eaa215fb63484dad09405b9867ae4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:45.682824 16207 tablet_bootstrap.cc:492] T 03eaa215fb63484dad09405b9867ae4d P 777856fcd5864cffa63bd364422fc3a8: Bootstrap starting.
I20260811 11:50:45.682963 16206 tablet_bootstrap.cc:492] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333: Bootstrap starting.
I20260811 11:50:45.683672 16207 tablet_bootstrap.cc:654] T 03eaa215fb63484dad09405b9867ae4d P 777856fcd5864cffa63bd364422fc3a8: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:45.683830 16206 tablet_bootstrap.cc:654] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:45.684525 16202 tablet_bootstrap.cc:492] T 03eaa215fb63484dad09405b9867ae4d P 09017061acfe48f6bb57b06e16ce084e: Bootstrap starting.
I20260811 11:50:45.684815 16207 tablet_bootstrap.cc:492] T 03eaa215fb63484dad09405b9867ae4d P 777856fcd5864cffa63bd364422fc3a8: No bootstrap required, opened a new log
I20260811 11:50:45.684908 16207 ts_tablet_manager.cc:1397] T 03eaa215fb63484dad09405b9867ae4d P 777856fcd5864cffa63bd364422fc3a8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:50:45.685308 16207 raft_consensus.cc:348] T 03eaa215fb63484dad09405b9867ae4d P 777856fcd5864cffa63bd364422fc3a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6065b7065aa84491a806b7276eb7b333" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45063 } } peers { permanent_uuid: "777856fcd5864cffa63bd364422fc3a8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44329 } } peers { permanent_uuid: "09017061acfe48f6bb57b06e16ce084e" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40795 } }
I20260811 11:50:45.685469 16207 raft_consensus.cc:374] T 03eaa215fb63484dad09405b9867ae4d P 777856fcd5864cffa63bd364422fc3a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:45.685491 16202 tablet_bootstrap.cc:654] T 03eaa215fb63484dad09405b9867ae4d P 09017061acfe48f6bb57b06e16ce084e: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:45.685526 16207 raft_consensus.cc:729] T 03eaa215fb63484dad09405b9867ae4d P 777856fcd5864cffa63bd364422fc3a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 777856fcd5864cffa63bd364422fc3a8, State: Initialized, Role: FOLLOWER
I20260811 11:50:45.685750 16206 tablet_bootstrap.cc:492] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333: No bootstrap required, opened a new log
I20260811 11:50:45.685701 16207 consensus_queue.cc:260] T 03eaa215fb63484dad09405b9867ae4d P 777856fcd5864cffa63bd364422fc3a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6065b7065aa84491a806b7276eb7b333" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45063 } } peers { permanent_uuid: "777856fcd5864cffa63bd364422fc3a8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44329 } } peers { permanent_uuid: "09017061acfe48f6bb57b06e16ce084e" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40795 } }
I20260811 11:50:45.685874 16206 ts_tablet_manager.cc:1397] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 11:50:45.686097 16207 ts_tablet_manager.cc:1428] T 03eaa215fb63484dad09405b9867ae4d P 777856fcd5864cffa63bd364422fc3a8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:45.686357 16206 raft_consensus.cc:348] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6065b7065aa84491a806b7276eb7b333" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45063 } } peers { permanent_uuid: "777856fcd5864cffa63bd364422fc3a8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44329 } } peers { permanent_uuid: "09017061acfe48f6bb57b06e16ce084e" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40795 } }
I20260811 11:50:45.686496 16206 raft_consensus.cc:374] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:45.686548 16206 raft_consensus.cc:729] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6065b7065aa84491a806b7276eb7b333, State: Initialized, Role: FOLLOWER
I20260811 11:50:45.686635 16202 tablet_bootstrap.cc:492] T 03eaa215fb63484dad09405b9867ae4d P 09017061acfe48f6bb57b06e16ce084e: No bootstrap required, opened a new log
I20260811 11:50:45.686743 16202 ts_tablet_manager.cc:1397] T 03eaa215fb63484dad09405b9867ae4d P 09017061acfe48f6bb57b06e16ce084e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 11:50:45.686692 16206 consensus_queue.cc:260] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6065b7065aa84491a806b7276eb7b333" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45063 } } peers { permanent_uuid: "777856fcd5864cffa63bd364422fc3a8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44329 } } peers { permanent_uuid: "09017061acfe48f6bb57b06e16ce084e" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40795 } }
I20260811 11:50:45.687058 16206 ts_tablet_manager.cc:1428] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:45.687216 16202 raft_consensus.cc:348] T 03eaa215fb63484dad09405b9867ae4d P 09017061acfe48f6bb57b06e16ce084e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6065b7065aa84491a806b7276eb7b333" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45063 } } peers { permanent_uuid: "777856fcd5864cffa63bd364422fc3a8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44329 } } peers { permanent_uuid: "09017061acfe48f6bb57b06e16ce084e" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40795 } }
I20260811 11:50:45.687371 16202 raft_consensus.cc:374] T 03eaa215fb63484dad09405b9867ae4d P 09017061acfe48f6bb57b06e16ce084e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:45.687428 16202 raft_consensus.cc:729] T 03eaa215fb63484dad09405b9867ae4d P 09017061acfe48f6bb57b06e16ce084e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09017061acfe48f6bb57b06e16ce084e, State: Initialized, Role: FOLLOWER
I20260811 11:50:45.687584 16202 consensus_queue.cc:260] T 03eaa215fb63484dad09405b9867ae4d P 09017061acfe48f6bb57b06e16ce084e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6065b7065aa84491a806b7276eb7b333" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45063 } } peers { permanent_uuid: "777856fcd5864cffa63bd364422fc3a8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44329 } } peers { permanent_uuid: "09017061acfe48f6bb57b06e16ce084e" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40795 } }
I20260811 11:50:45.688158 16202 ts_tablet_manager.cc:1428] T 03eaa215fb63484dad09405b9867ae4d P 09017061acfe48f6bb57b06e16ce084e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:45.688331 16172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.688041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.65:41385)
0811 11:50:45.688195 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:45.688257 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260811 11:50:45.689159 16171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.688568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.66:40489)
0811 11:50:45.689037 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:45.689093 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375}
I20260811 11:50:45.689786 16173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.689546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.67:37051)
0811 11:50:45.689662 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:45.689716 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260811 11:50:45.690742 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.690242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.66:52359)
0811 11:50:45.690531 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:45.690617 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":109,"threads_started":1}
I20260811 11:50:45.690750 15816 heartbeater.cc:503] Master 127.3.221.125:39327 was elected leader, sending a full tablet report...
I20260811 11:50:45.693645 16173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.693363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.67:52319)
0811 11:50:45.693516 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:45.693570 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260811 11:50:45.694621 16039 heartbeater.cc:503] Master 127.3.221.125:39327 was elected leader, sending a full tablet report...
I20260811 11:50:45.695339 16172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.695099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.65:42837)
0811 11:50:45.695204 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:45.695264 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260811 11:50:45.695531 15594 heartbeater.cc:503] Master 127.3.221.125:39327 was elected leader, sending a full tablet report...
I20260811 11:50:45.957201 16215 raft_consensus.cc:482] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:50:45.957432 16215 raft_consensus.cc:504] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6065b7065aa84491a806b7276eb7b333" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45063 } } peers { permanent_uuid: "777856fcd5864cffa63bd364422fc3a8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44329 } } peers { permanent_uuid: "09017061acfe48f6bb57b06e16ce084e" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40795 } }
I20260811 11:50:45.958537 16215 leader_election.cc:288] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 777856fcd5864cffa63bd364422fc3a8 (127.3.221.67:44329), 09017061acfe48f6bb57b06e16ce084e (127.3.221.66:40795)
I20260811 11:50:45.963526 16172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.958791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:44329 (local address 127.3.221.65:34001)
0811 11:50:45.958991 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:45.959007 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:45.959168 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:45.959441 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:45.959444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:45.959459 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:45.959658 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:45.959668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:45.960674 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:45.960677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:45.962142 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:45.962151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:45.963055 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:45.963058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:45.963227 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:45.963275 (+ 48us) client_negotiation.cc:769] Sending connection context
0811 11:50:45.963339 (+ 64us) client_negotiation.cc:240] Negotiation successful
0811 11:50:45.963391 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":159}
I20260811 11:50:45.963547 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.958899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:34001 (local address 127.3.221.67:44329)
0811 11:50:45.959009 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:45.959013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:45.959123 (+ 110us) server_negotiation.cc:407] Connection header received
0811 11:50:45.959275 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:45.959278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:45.959337 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:45.959424 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:45.959768 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:45.960553 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:45.962275 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:45.962898 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:45.963017 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:45.963071 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:45.963366 (+ 295us) server_negotiation.cc:299] Negotiation successful
0811 11:50:45.963441 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 11:50:45.963547 16200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.958933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:54835 (local address 127.3.221.66:40795)
0811 11:50:45.959062 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:45.959066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:45.959284 (+ 218us) server_negotiation.cc:407] Connection header received
0811 11:50:45.959369 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:45.959372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:45.959422 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:45.959492 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:45.959768 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:45.960571 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:45.962275 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:45.962892 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:45.963017 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:45.963071 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:45.963336 (+ 265us) server_negotiation.cc:299] Negotiation successful
0811 11:50:45.963418 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 11:50:45.963536 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.958843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:40795 (local address 127.3.221.65:54835)
0811 11:50:45.959186 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:45.959198 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:45.959294 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:45.959507 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:45.959510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:45.959522 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:45.959666 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:45.959672 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:45.960674 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:45.960677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:45.962138 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:45.962151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:45.963044 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:45.963049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:45.963162 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:45.963206 (+ 44us) client_negotiation.cc:769] Sending connection context
0811 11:50:45.963282 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 11:50:45.963364 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":103,"thread_start_us":138,"threads_started":1}
I20260811 11:50:45.964126 15962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03eaa215fb63484dad09405b9867ae4d" candidate_uuid: "6065b7065aa84491a806b7276eb7b333" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "777856fcd5864cffa63bd364422fc3a8" is_pre_election: true
I20260811 11:50:45.964262 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03eaa215fb63484dad09405b9867ae4d" candidate_uuid: "6065b7065aa84491a806b7276eb7b333" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09017061acfe48f6bb57b06e16ce084e" is_pre_election: true
I20260811 11:50:45.964396 15737 raft_consensus.cc:2454] T 03eaa215fb63484dad09405b9867ae4d P 09017061acfe48f6bb57b06e16ce084e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6065b7065aa84491a806b7276eb7b333 in term 0.
I20260811 11:50:45.964411 15962 raft_consensus.cc:2454] T 03eaa215fb63484dad09405b9867ae4d P 777856fcd5864cffa63bd364422fc3a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6065b7065aa84491a806b7276eb7b333 in term 0.
I20260811 11:50:45.964805 15404 leader_election.cc:302] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6065b7065aa84491a806b7276eb7b333, 777856fcd5864cffa63bd364422fc3a8; no voters:
I20260811 11:50:45.965109 16215 raft_consensus.cc:2781] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:50:45.965178 16215 raft_consensus.cc:482] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:50:45.965205 16215 raft_consensus.cc:3037] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:45.966288 16215 raft_consensus.cc:504] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6065b7065aa84491a806b7276eb7b333" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45063 } } peers { permanent_uuid: "777856fcd5864cffa63bd364422fc3a8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44329 } } peers { permanent_uuid: "09017061acfe48f6bb57b06e16ce084e" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40795 } }
I20260811 11:50:45.966670 16215 leader_election.cc:288] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333 [CANDIDATE]: Term 1 election: Requested vote from peers 777856fcd5864cffa63bd364422fc3a8 (127.3.221.67:44329), 09017061acfe48f6bb57b06e16ce084e (127.3.221.66:40795)
I20260811 11:50:45.967077 15962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03eaa215fb63484dad09405b9867ae4d" candidate_uuid: "6065b7065aa84491a806b7276eb7b333" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "777856fcd5864cffa63bd364422fc3a8"
I20260811 11:50:45.967103 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03eaa215fb63484dad09405b9867ae4d" candidate_uuid: "6065b7065aa84491a806b7276eb7b333" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09017061acfe48f6bb57b06e16ce084e"
I20260811 11:50:45.967242 15962 raft_consensus.cc:3037] T 03eaa215fb63484dad09405b9867ae4d P 777856fcd5864cffa63bd364422fc3a8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:45.967279 15737 raft_consensus.cc:3037] T 03eaa215fb63484dad09405b9867ae4d P 09017061acfe48f6bb57b06e16ce084e [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:45.968283 15962 raft_consensus.cc:2454] T 03eaa215fb63484dad09405b9867ae4d P 777856fcd5864cffa63bd364422fc3a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6065b7065aa84491a806b7276eb7b333 in term 1.
I20260811 11:50:45.968281 15737 raft_consensus.cc:2454] T 03eaa215fb63484dad09405b9867ae4d P 09017061acfe48f6bb57b06e16ce084e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6065b7065aa84491a806b7276eb7b333 in term 1.
I20260811 11:50:45.968601 15404 leader_election.cc:302] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6065b7065aa84491a806b7276eb7b333, 777856fcd5864cffa63bd364422fc3a8; no voters:
I20260811 11:50:45.968786 16215 raft_consensus.cc:2781] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:45.969017 16215 raft_consensus.cc:686] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333 [term 1 LEADER]: Becoming Leader. State: Replica: 6065b7065aa84491a806b7276eb7b333, State: Running, Role: LEADER
I20260811 11:50:45.969319 16215 consensus_queue.cc:237] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6065b7065aa84491a806b7276eb7b333" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45063 } } peers { permanent_uuid: "777856fcd5864cffa63bd364422fc3a8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44329 } } peers { permanent_uuid: "09017061acfe48f6bb57b06e16ce084e" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40795 } }
I20260811 11:50:45.970019 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:45.969726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.65:40093)
0811 11:50:45.969876 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:45.969941 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260811 11:50:45.972019 14983 catalog_manager.cc:5495] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6065b7065aa84491a806b7276eb7b333 (127.3.221.65). New cstate: current_term: 1 leader_uuid: "6065b7065aa84491a806b7276eb7b333" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6065b7065aa84491a806b7276eb7b333" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "777856fcd5864cffa63bd364422fc3a8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09017061acfe48f6bb57b06e16ce084e" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40795 } health_report { overall_health: UNKNOWN } } }
I20260811 11:50:46.048527 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:46.045060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33662 (local address 127.3.221.65:45063)
0811 11:50:46.045195 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:46.045198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:46.045242 (+ 44us) server_negotiation.cc:407] Connection header received
0811 11:50:46.045379 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:46.045381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:46.045441 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:46.045531 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:46.045747 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:46.046262 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:46.047141 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:46.047371 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:46.047450 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:46.047715 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:46.048196 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:46.048297 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:46.048351 (+ 54us) server_negotiation.cc:299] Negotiation successful
0811 11:50:46.048408 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 11:50:46.051569 15737 raft_consensus.cc:1261] T 03eaa215fb63484dad09405b9867ae4d P 09017061acfe48f6bb57b06e16ce084e [term 1 FOLLOWER]: Refusing update from remote peer 6065b7065aa84491a806b7276eb7b333: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:46.051841 15962 raft_consensus.cc:1261] T 03eaa215fb63484dad09405b9867ae4d P 777856fcd5864cffa63bd364422fc3a8 [term 1 FOLLOWER]: Refusing update from remote peer 6065b7065aa84491a806b7276eb7b333: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:46.052129 16215 consensus_queue.cc:1035] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09017061acfe48f6bb57b06e16ce084e" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:46.052479 16221 consensus_queue.cc:1035] T 03eaa215fb63484dad09405b9867ae4d P 6065b7065aa84491a806b7276eb7b333 [LEADER]: Connected to new peer: Peer: permanent_uuid: "777856fcd5864cffa63bd364422fc3a8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:46.053889 16171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:46.053628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.66:41163)
0811 11:50:46.053738 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:46.053807 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260811 11:50:46.054193 16173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:46.053918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.67:51039)
0811 11:50:46.054072 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:46.054127 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260811 11:50:46.058360 16173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:46.058042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.67:41285)
0811 11:50:46.058221 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:46.058281 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260811 11:50:46.058557 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:46.058282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.65:48351)
0811 11:50:46.058431 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:46.058484 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260811 11:50:46.060237 16233 mvcc.cc:205] Tried to move back new op lower bound from 7317295292622024704 to 7317295292293574656. Current Snapshot: MvccSnapshot[applied={T|T < 7317295292622024704}]
I20260811 11:50:46.060967 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:46.060665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.65:40065)
0811 11:50:46.060833 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:46.060892 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260811 11:50:46.063184 16171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:46.062856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46443 (local address 127.3.221.66:43193)
0811 11:50:46.063050 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:46.063106 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260811 11:50:46.065872 14982 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:46.066021 14982 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:46.070032 14982 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30ce91da8cce49dba59fb1865beb961c: Sending DeleteTablet for 3 replicas of tablet 03eaa215fb63484dad09405b9867ae4d
I20260811 11:50:46.070861 15922 tablet_service.cc:1514] Processing DeleteTablet for tablet 03eaa215fb63484dad09405b9867ae4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:46 UTC) from {username='oryx', principal='oryx/127.3.221.125@KRBTEST.COM'} at 127.0.0.1:57534
I20260811 11:50:46.070912 15700 tablet_service.cc:1514] Processing DeleteTablet for tablet 03eaa215fb63484dad09405b9867ae4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:46 UTC) from {username='oryx', principal='oryx/127.3.221.125@KRBTEST.COM'} at 127.0.0.1:50604
I20260811 11:50:46.071359 15475 tablet_service.cc:1514] Processing DeleteTablet for tablet 03eaa215fb63484dad09405b9867ae4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:46 UTC) from {username='oryx', principal='oryx/127.3.221.125@KRBTEST.COM'} at 127.0.0.1:33648
I20260811 11:50:46.071738 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 15215
I20260811 11:50:46.072372 16248 tablet_replica.cc:330] T 03eaa215fb63484dad09405b9867ae4d P 09017061acfe48f6bb57b06e16ce084e: stopping tablet replica
I20260811 11:50:46.072533 16248 raft_consensus.cc:2229] T 03eaa215fb63484dad09405b9867ae4d P 09017061acfe48f6bb57b06e16ce084e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:46.072702 16248 raft_consensus.cc:2258] T 03eaa215fb63484dad09405b9867ae4d P 09017061acfe48f6bb57b06e16ce084e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:46.078370 16248 ts_tablet_manager.cc:1905] T 03eaa215fb63484dad09405b9867ae4d P 09017061acfe48f6bb57b06e16ce084e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:46.081789 16248 ts_tablet_manager.cc:1918] T 03eaa215fb63484dad09405b9867ae4d P 09017061acfe48f6bb57b06e16ce084e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:50:46.081907 16248 log.cc:1196] T 03eaa215fb63484dad09405b9867ae4d P 09017061acfe48f6bb57b06e16ce084e: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/03eaa215fb63484dad09405b9867ae4d
I20260811 11:50:46.082155 16248 ts_tablet_manager.cc:1939] T 03eaa215fb63484dad09405b9867ae4d P 09017061acfe48f6bb57b06e16ce084e: Deleting consensus metadata
I20260811 11:50:46.083000 14959 catalog_manager.cc:4841] TS 09017061acfe48f6bb57b06e16ce084e (127.3.221.66:40795): tablet 03eaa215fb63484dad09405b9867ae4d (table test-table [id=b7e7b690770f44d1aff901b235d06950]) successfully deleted
I20260811 11:50:46.085096 16246 tablet_replica.cc:330] T 03eaa215fb63484dad09405b9867ae4d P 777856fcd5864cffa63bd364422fc3a8: stopping tablet replica
I20260811 11:50:46.085281 16246 raft_consensus.cc:2229] T 03eaa215fb63484dad09405b9867ae4d P 777856fcd5864cffa63bd364422fc3a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:46.085451 16246 raft_consensus.cc:2258] T 03eaa215fb63484dad09405b9867ae4d P 777856fcd5864cffa63bd364422fc3a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:46.086247 16246 ts_tablet_manager.cc:1905] T 03eaa215fb63484dad09405b9867ae4d P 777856fcd5864cffa63bd364422fc3a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:46.088363 16246 ts_tablet_manager.cc:1918] T 03eaa215fb63484dad09405b9867ae4d P 777856fcd5864cffa63bd364422fc3a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:50:46.088442 16246 log.cc:1196] T 03eaa215fb63484dad09405b9867ae4d P 777856fcd5864cffa63bd364422fc3a8: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/03eaa215fb63484dad09405b9867ae4d
I20260811 11:50:46.088675 16246 ts_tablet_manager.cc:1939] T 03eaa215fb63484dad09405b9867ae4d P 777856fcd5864cffa63bd364422fc3a8: Deleting consensus metadata
I20260811 11:50:46.089565 14961 catalog_manager.cc:4841] TS 777856fcd5864cffa63bd364422fc3a8 (127.3.221.67:44329): tablet 03eaa215fb63484dad09405b9867ae4d (table test-table [id=b7e7b690770f44d1aff901b235d06950]) successfully deleted
I20260811 11:50:46.092350 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 15601
I20260811 11:50:46.092401 16197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:46.092122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:45063 (local address 127.0.0.1:33674)
0811 11:50:46.092249 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:46.092306 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.65:45063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260811 11:50:46.101135 14963 catalog_manager.cc:4584] TS 6065b7065aa84491a806b7276eb7b333 (127.3.221.65:45063): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 03eaa215fb63484dad09405b9867ae4d: Network error: Client connection negotiation failed: client connection to 127.3.221.65:45063: connect: Connection refused (error 111)
I20260811 11:50:46.109462 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 15823
I20260811 11:50:46.121692 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 14775
I20260811 11:50:46.132544 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 14944
2026-08-11T11:50:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7844 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:50:46 dist-test-slave-bndk krb5kdc[16255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:50:46 dist-test-slave-bndk krb5kdc[16255](info): set up 2 sockets
Aug 11 11:50:46 dist-test-slave-bndk krb5kdc[16255](info): commencing operation
krb5kdc: starting...
W20260811 11:50:47.312469 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:50:47 dist-test-slave-bndk krb5kdc[16255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:50:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:50:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.126.keytab.
Entry for principal oryx/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.126.keytab.
I20260811 11:50:47.446365 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:46025
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.126.keytab
--principal=oryx/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:35587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:46025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:47.562366 16273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:47.562675 16273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:47.562734 16273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:47.566879 16273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:50:47.566949 16273 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 11:50:47.566974 16273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:47.566998 16273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:47.567027 16273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:50:47.567056 16273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:50:47.572633 16273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:35587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:46025
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.126.keytab
--principal=oryx/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:46025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:47.573927 16273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:47.575018 16273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:47.584416 16281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:47.584658 16273 server_base.cc:1034] running on GCE node
W20260811 11:50:47.584455 16278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:47.584734 16279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:47.585413 16273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:47.586653 16273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:47.587855 16273 hybrid_clock.cc:648] HybridClock initialized: now 1786449047587838 us; error 29 us; skew 500 ppm
Aug 11 11:50:47 dist-test-slave-bndk krb5kdc[16255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449047, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:47.590879 16273 init.cc:377] Logged in from keytab as oryx/127.3.221.126@KRBTEST.COM (short username oryx)
I20260811 11:50:47.591887 16273 webserver.cc:460] Webserver started at http://127.3.221.126:46625/ using document root <none> and password file <none>
I20260811 11:50:47.592468 16273 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:47.592540 16273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:47.592765 16273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:47.595017 16273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "7c4781c123434fb1912feba4dd41bae9"
format_stamp: "Formatted at 2026-08-11 11:50:47 on dist-test-slave-bndk"
server_key: "2bb630b40a6c098e2ac742f6d822ad8f"
server_key_iv: "41246f7e7db640822b9ef858da1585b4"
server_key_version: "encryptionkey@0"
I20260811 11:50:47.595592 16273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "7c4781c123434fb1912feba4dd41bae9"
format_stamp: "Formatted at 2026-08-11 11:50:47 on dist-test-slave-bndk"
server_key: "2bb630b40a6c098e2ac742f6d822ad8f"
server_key_iv: "41246f7e7db640822b9ef858da1585b4"
server_key_version: "encryptionkey@0"
I20260811 11:50:47.599622 16273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260811 11:50:47.602356 16296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:47.603820 16273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:50:47.603976 16273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "7c4781c123434fb1912feba4dd41bae9"
format_stamp: "Formatted at 2026-08-11 11:50:47 on dist-test-slave-bndk"
server_key: "2bb630b40a6c098e2ac742f6d822ad8f"
server_key_iv: "41246f7e7db640822b9ef858da1585b4"
server_key_version: "encryptionkey@0"
I20260811 11:50:47.604094 16273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:47.616725 16273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:47.619822 16273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:47.620008 16273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:47.627950 16273 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:46025
I20260811 11:50:47.628005 16404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:46025 every 8 connection(s)
I20260811 11:50:47.628966 16273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:50:47.631940 16405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:47.633993 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 16273
I20260811 11:50:47.634109 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:50:47.634421 3957 external_mini_cluster.cc:1428] Setting key 019c1a9e204623a400ed68dcf20887a5
I20260811 11:50:47.638705 16405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9: Bootstrap starting.
I20260811 11:50:47.641553 16405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:47.642511 16405 log.cc:824] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9: Log is configured to *not* fsync() on all Append() calls
Aug 11 11:50:47 dist-test-slave-bndk krb5kdc[16255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM
I20260811 11:50:47.644959 16405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9: No bootstrap required, opened a new log
I20260811 11:50:47.649415 16405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4781c123434fb1912feba4dd41bae9" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46025 } }
I20260811 11:50:47.649794 16405 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:47.649889 16405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c4781c123434fb1912feba4dd41bae9, State: Initialized, Role: FOLLOWER
I20260811 11:50:47.650422 16405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4781c123434fb1912feba4dd41bae9" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46025 } }
I20260811 11:50:47.650592 16405 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:50:47.650660 16405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:50:47.650758 16405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:47.651072 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:47.635931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53224 (local address 127.3.221.126:46025)
0811 11:50:47.636417 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:47.636438 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:47.636483 (+ 45us) server_negotiation.cc:407] Connection header received
0811 11:50:47.637429 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:47.637456 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:47.637918 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:47.638303 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:47.639063 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:47.640561 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:47.641640 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:47.642342 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:47.642539 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:47.645337 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:47.645345 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:47.645373 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:47.645413 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:47.647511 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:47.647976 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:47.647982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:47.647989 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:47.648157 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:47.648463 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:47.648467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:47.648469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:47.649036 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:47.649279 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:47.649609 (+ 330us) server_negotiation.cc:299] Negotiation successful
0811 11:50:47.650019 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":141,"threads_started":1}
I20260811 11:50:47.651868 16405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4781c123434fb1912feba4dd41bae9" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46025 } }
I20260811 11:50:47.652235 16405 leader_election.cc:302] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c4781c123434fb1912feba4dd41bae9; no voters:
I20260811 11:50:47.652591 16405 leader_election.cc:288] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:50:47.652794 16414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:47.653020 16414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9 [term 1 LEADER]: Becoming Leader. State: Replica: 7c4781c123434fb1912feba4dd41bae9, State: Running, Role: LEADER
I20260811 11:50:47.653432 16414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4781c123434fb1912feba4dd41bae9" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46025 } }
I20260811 11:50:47.653944 16405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:50:47.655169 16415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c4781c123434fb1912feba4dd41bae9. Latest consensus state: current_term: 1 leader_uuid: "7c4781c123434fb1912feba4dd41bae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4781c123434fb1912feba4dd41bae9" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46025 } } }
I20260811 11:50:47.655225 16418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c4781c123434fb1912feba4dd41bae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4781c123434fb1912feba4dd41bae9" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46025 } } }
I20260811 11:50:47.655306 16415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:47.655323 16418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9 [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:47.655740 16431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:50:47.658839 16431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:50:47.664959 16431 catalog_manager.cc:1347] Generated new cluster ID: ecbe2238a7b948a3b85b24cc4ca32ec9
I20260811 11:50:47.665037 16431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:50:47.674335 16431 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:50:47.675336 16431 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:50:47.681648 16431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9: Generated new TSK 0
I20260811 11:50:47.682575 16431 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.65.keytab.
Entry for principal oryx/127.3.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.65.keytab.
I20260811 11:50:47.716737 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:46025
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.65.keytab
--principal=oryx/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:35587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:47.837605 16450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:47.837901 16450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:47.837947 16450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:47.842051 16450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:47.842127 16450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:47.842167 16450 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 11:50:47.842218 16450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:50:47.847750 16450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:35587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.65.keytab
--principal=oryx/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.221.126:46025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:47.848907 16450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:47.849972 16450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:50:47.860335 16450 server_base.cc:1034] running on GCE node
W20260811 11:50:47.860211 16459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:47.860222 16466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:47.860255 16457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:47.860965 16450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:47.861701 16450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:47.862866 16450 hybrid_clock.cc:648] HybridClock initialized: now 1786449047862844 us; error 36 us; skew 500 ppm
Aug 11 11:50:47 dist-test-slave-bndk krb5kdc[16255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449047, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:47.865859 16450 init.cc:377] Logged in from keytab as oryx/127.3.221.65@KRBTEST.COM (short username oryx)
I20260811 11:50:47.866840 16450 webserver.cc:460] Webserver started at http://127.3.221.65:43695/ using document root <none> and password file <none>
I20260811 11:50:47.867414 16450 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:47.867488 16450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:47.867713 16450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:47.869923 16450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "b457496f7b59437eaec5d6efdd22b306"
format_stamp: "Formatted at 2026-08-11 11:50:47 on dist-test-slave-bndk"
server_key: "3291bdb615c5b73bdbf0c773dba28bb2"
server_key_iv: "8fa257378ac819e7d67ec91812f85f1a"
server_key_version: "encryptionkey@0"
I20260811 11:50:47.870512 16450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "b457496f7b59437eaec5d6efdd22b306"
format_stamp: "Formatted at 2026-08-11 11:50:47 on dist-test-slave-bndk"
server_key: "3291bdb615c5b73bdbf0c773dba28bb2"
server_key_iv: "8fa257378ac819e7d67ec91812f85f1a"
server_key_version: "encryptionkey@0"
I20260811 11:50:47.874583 16450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260811 11:50:47.877310 16475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:47.878695 16450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:50:47.878845 16450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "b457496f7b59437eaec5d6efdd22b306"
format_stamp: "Formatted at 2026-08-11 11:50:47 on dist-test-slave-bndk"
server_key: "3291bdb615c5b73bdbf0c773dba28bb2"
server_key_iv: "8fa257378ac819e7d67ec91812f85f1a"
server_key_version: "encryptionkey@0"
I20260811 11:50:47.878962 16450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:47.891101 16450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:47.894387 16450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:47.894582 16450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:47.896201 16450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:47.896257 16450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:47.896313 16450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:47.896340 16450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 11:50:47 dist-test-slave-bndk krb5kdc[16255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449047, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM
I20260811 11:50:47.906929 16450 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:45263
I20260811 11:50:47.906971 16726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:45263 every 8 connection(s)
I20260811 11:50:47.908012 16450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:50:47.910604 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:47.897005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:43771 (local address 127.3.221.126:46025)
0811 11:50:47.897130 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:47.897135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:47.898347 (+ 1212us) server_negotiation.cc:407] Connection header received
0811 11:50:47.899508 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:47.899512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:47.899576 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:47.899635 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:47.901505 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:47.902332 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:47.903636 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:47.903886 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:47.903977 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:47.906714 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:47.906721 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:47.906724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:47.906780 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:47.908377 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:47.909010 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:47.909014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:47.909016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:47.909104 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:47.909488 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:47.909491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:47.909493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:47.909694 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:47.909786 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:47.910290 (+ 504us) server_negotiation.cc:299] Negotiation successful
0811 11:50:47.910434 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 11:50:47.911410 16493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:47.897655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46025 (local address 127.3.221.65:43771)
0811 11:50:47.898173 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:47.898218 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:47.899239 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:47.900085 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:47.900095 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:47.900605 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:47.901337 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:47.901354 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:47.902481 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:47.902485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:47.903506 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:47.903517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:47.904039 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:47.904043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:47.904222 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:47.904806 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:47.904834 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:47.906492 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:47.908546 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:47.908552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:47.908568 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:47.908901 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:47.909189 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:47.909192 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:47.909194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:47.909387 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:47.909795 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:47.909802 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:47.910052 (+ 250us) client_negotiation.cc:769] Sending connection context
0811 11:50:47.910211 (+ 159us) client_negotiation.cc:240] Negotiation successful
0811 11:50:47.910563 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":144,"threads_started":1}
I20260811 11:50:47.913197 16728 heartbeater.cc:347] Connected to a master server at 127.3.221.126:46025
I20260811 11:50:47.913409 16728 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:47.914115 16728 heartbeater.cc:511] Master 127.3.221.126:46025 requested a full tablet report, sending...
I20260811 11:50:47.915213 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 16450
I20260811 11:50:47.915373 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:50:47.915743 3957 external_mini_cluster.cc:1428] Setting key 18bb979c3fef9d11f1daed59f188a198
I20260811 11:50:47.915934 16320 ts_manager.cc:194] Registered new tserver with Master: b457496f7b59437eaec5d6efdd22b306 (127.3.221.65:45263)
I20260811 11:50:47.917431 16320 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:43771
I20260811 11:50:47.923432 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:47.916707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:59415 (local address 127.3.221.126:46025)
0811 11:50:47.916830 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:47.916834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:47.916846 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:50:47.916876 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:47.916878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:47.916911 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:47.916991 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:47.917563 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:47.918228 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:47.919148 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:47.919378 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:47.919466 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:47.920560 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:47.920564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:47.920567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:47.920592 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:47.921889 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:47.922282 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:47.922286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:47.922288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:47.922368 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:47.922625 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:47.922628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:47.922630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:47.922810 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:47.922914 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:47.923168 (+ 254us) server_negotiation.cc:299] Negotiation successful
0811 11:50:47.923278 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 11:50:47.923452 16740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:47.915438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46025 (local address 127.3.221.65:59415)
0811 11:50:47.915759 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:47.915775 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:47.915913 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:47.917020 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:47.917023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:47.917286 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:47.917426 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:47.917434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:47.918369 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:47.918371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:47.919032 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:47.919042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:47.919483 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:47.919487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:47.919578 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:47.920064 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:47.920077 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:47.920427 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:47.921992 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:47.921996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:47.921998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:47.922193 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:47.922452 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:47.922455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:47.922457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:47.922543 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:47.922916 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:47.922919 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:47.923014 (+ 95us) client_negotiation.cc:769] Sending connection context
0811 11:50:47.923102 (+ 88us) client_negotiation.cc:240] Negotiation successful
0811 11:50:47.923254 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for oryx/127.3.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.66.keytab.
Entry for principal oryx/127.3.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.66.keytab.
I20260811 11:50:47.948163 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:46025
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.66.keytab
--principal=oryx/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:35587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:48.065304 16743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:48.065596 16743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:48.065649 16743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:48.069756 16743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:48.069834 16743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:48.069877 16743 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 11:50:48.069929 16743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:50:48.075663 16743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:35587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.66.keytab
--principal=oryx/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.221.126:46025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:48.076800 16743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:48.077891 16743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:48.088027 16754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:48.088037 16752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:48.088043 16750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:48.088182 16743 server_base.cc:1034] running on GCE node
I20260811 11:50:48.088649 16743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:48.089330 16743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:48.090517 16743 hybrid_clock.cc:648] HybridClock initialized: now 1786449048090492 us; error 34 us; skew 500 ppm
Aug 11 11:50:48 dist-test-slave-bndk krb5kdc[16255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449048, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:48.093403 16743 init.cc:377] Logged in from keytab as oryx/127.3.221.66@KRBTEST.COM (short username oryx)
I20260811 11:50:48.094316 16743 webserver.cc:460] Webserver started at http://127.3.221.66:44281/ using document root <none> and password file <none>
I20260811 11:50:48.094898 16743 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:48.094971 16743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:48.095193 16743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:48.097409 16743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "4df141eeadc64b96ad0d38bd94281144"
format_stamp: "Formatted at 2026-08-11 11:50:48 on dist-test-slave-bndk"
server_key: "dc4b3aa11c5010865b04034e078c5535"
server_key_iv: "0c1b35b39ae05930db6ce1420c05b0ef"
server_key_version: "encryptionkey@0"
I20260811 11:50:48.097980 16743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "4df141eeadc64b96ad0d38bd94281144"
format_stamp: "Formatted at 2026-08-11 11:50:48 on dist-test-slave-bndk"
server_key: "dc4b3aa11c5010865b04034e078c5535"
server_key_iv: "0c1b35b39ae05930db6ce1420c05b0ef"
server_key_version: "encryptionkey@0"
I20260811 11:50:48.102031 16743 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:50:48.104743 16767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:48.106132 16743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:50:48.106266 16743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "4df141eeadc64b96ad0d38bd94281144"
format_stamp: "Formatted at 2026-08-11 11:50:48 on dist-test-slave-bndk"
server_key: "dc4b3aa11c5010865b04034e078c5535"
server_key_iv: "0c1b35b39ae05930db6ce1420c05b0ef"
server_key_version: "encryptionkey@0"
I20260811 11:50:48.106386 16743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:48.125190 16743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:48.128599 16743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:48.128806 16743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:48.130426 16743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:48.130483 16743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:48.130538 16743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:48.130568 16743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 11:50:48 dist-test-slave-bndk krb5kdc[16255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449048, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM
I20260811 11:50:48.141371 16743 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:44233
I20260811 11:50:48.141399 16949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:44233 every 8 connection(s)
I20260811 11:50:48.142338 16743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:50:48.145469 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.131340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:57547 (local address 127.3.221.126:46025)
0811 11:50:48.131471 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:48.131475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:48.132601 (+ 1126us) server_negotiation.cc:407] Connection header received
0811 11:50:48.133727 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:48.133730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:48.133791 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:48.133884 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:48.135544 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.136433 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.138282 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.138533 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.138627 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.141970 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:48.141973 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:48.141975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:48.142004 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:48.143586 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:48.144126 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:48.144129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:48.144130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:48.144217 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:48.144501 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:48.144503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:48.144504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:48.144685 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:48.144759 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:48.145188 (+ 429us) server_negotiation.cc:299] Negotiation successful
0811 11:50:48.145344 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 11:50:48.146148 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 16743
I20260811 11:50:48.146268 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:50:48.146327 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.131921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46025 (local address 127.3.221.66:57547)
0811 11:50:48.132441 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:48.132487 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:48.133504 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:48.134128 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:48.134138 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:48.134631 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:48.135367 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.135389 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.136585 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.136589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.138101 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.138116 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.138717 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.138722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.138940 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.139786 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:48.139811 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:48.141769 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:48.143694 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:48.143698 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:48.143710 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:48.144007 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:48.144286 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:48.144288 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:48.144290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:48.144420 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:48.144756 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:48.144761 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:48.144995 (+ 234us) client_negotiation.cc:769] Sending connection context
0811 11:50:48.145144 (+ 149us) client_negotiation.cc:240] Negotiation successful
0811 11:50:48.145456 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":152,"threads_started":1}
I20260811 11:50:48.146592 3957 external_mini_cluster.cc:1428] Setting key f661108b367a3aac712e29642da67f1f
I20260811 11:50:48.147671 16950 heartbeater.cc:347] Connected to a master server at 127.3.221.126:46025
I20260811 11:50:48.147823 16950 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:48.148471 16950 heartbeater.cc:511] Master 127.3.221.126:46025 requested a full tablet report, sending...
I20260811 11:50:48.149637 16313 ts_manager.cc:194] Registered new tserver with Master: 4df141eeadc64b96ad0d38bd94281144 (127.3.221.66:44233)
I20260811 11:50:48.150384 16313 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:57547
I20260811 11:50:48.158303 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.151088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:53567 (local address 127.3.221.126:46025)
0811 11:50:48.151190 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:48.151194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:48.151207 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:50:48.151256 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:48.151258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:48.151291 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:48.151358 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:48.151879 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.152602 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.153671 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.153878 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.153951 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.155176 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:48.155180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:48.155183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:48.155212 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:48.156698 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:48.157153 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:48.157155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:48.157157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:48.157265 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:48.157520 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:48.157522 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:48.157523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:48.157733 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:48.157820 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:48.158086 (+ 266us) server_negotiation.cc:299] Negotiation successful
0811 11:50:48.158180 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260811 11:50:48.158409 16959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.149880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46025 (local address 127.3.221.66:53567)
0811 11:50:48.150658 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:48.150673 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:48.150762 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:48.151399 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:48.151401 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:48.151628 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:48.151748 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.151756 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.152738 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.152741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.153573 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.153586 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.153981 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.153985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.154074 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.154576 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:48.154591 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:48.155055 (+ 464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:48.156810 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:48.156814 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:48.156816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:48.157064 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:48.157336 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:48.157340 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:48.157342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:48.157438 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:48.157823 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:48.157826 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:48.157936 (+ 110us) client_negotiation.cc:769] Sending connection context
0811 11:50:48.158049 (+ 113us) client_negotiation.cc:240] Negotiation successful
0811 11:50:48.158197 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for oryx/127.3.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.67.keytab.
Entry for principal oryx/127.3.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.67.keytab.
I20260811 11:50:48.178732 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:46025
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.67.keytab
--principal=oryx/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:35587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:48.293624 16963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:48.293900 16963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:48.293946 16963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:48.297992 16963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:48.298059 16963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:48.298099 16963 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 11:50:48.298134 16963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:50:48.303489 16963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:35587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.67.keytab
--principal=oryx/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.221.126:46025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:48.304593 16963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:48.305729 16963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:48.316571 16975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:48.316565 16974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:48.316716 16963 server_base.cc:1034] running on GCE node
W20260811 11:50:48.316566 16977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:48.317368 16963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:48.318056 16963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:48.319250 16963 hybrid_clock.cc:648] HybridClock initialized: now 1786449048319225 us; error 36 us; skew 500 ppm
Aug 11 11:50:48 dist-test-slave-bndk krb5kdc[16255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449048, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:48.327486 16963 init.cc:377] Logged in from keytab as oryx/127.3.221.67@KRBTEST.COM (short username oryx)
I20260811 11:50:48.328504 16963 webserver.cc:460] Webserver started at http://127.3.221.67:41563/ using document root <none> and password file <none>
I20260811 11:50:48.329093 16963 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:48.329159 16963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:48.329411 16963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:48.331583 16963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "fba11c35b7c549fd85bf67db6558f280"
format_stamp: "Formatted at 2026-08-11 11:50:48 on dist-test-slave-bndk"
server_key: "0421363120bd603e0c9647a49a28601d"
server_key_iv: "d5e2d098f2292551737701688866919c"
server_key_version: "encryptionkey@0"
I20260811 11:50:48.332134 16963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "fba11c35b7c549fd85bf67db6558f280"
format_stamp: "Formatted at 2026-08-11 11:50:48 on dist-test-slave-bndk"
server_key: "0421363120bd603e0c9647a49a28601d"
server_key_iv: "d5e2d098f2292551737701688866919c"
server_key_version: "encryptionkey@0"
I20260811 11:50:48.336243 16963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 11:50:48.338848 16988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:48.340116 16963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 11:50:48.340225 16963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "fba11c35b7c549fd85bf67db6558f280"
format_stamp: "Formatted at 2026-08-11 11:50:48 on dist-test-slave-bndk"
server_key: "0421363120bd603e0c9647a49a28601d"
server_key_iv: "d5e2d098f2292551737701688866919c"
server_key_version: "encryptionkey@0"
I20260811 11:50:48.340346 16963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:48.352837 16963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:48.355975 16963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:48.356139 16963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:48.357692 16963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:48.357750 16963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:48.357806 16963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:48.357836 16963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 11:50:48 dist-test-slave-bndk krb5kdc[16255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449048, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM
I20260811 11:50:48.368417 16963 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:37619
I20260811 11:50:48.368523 17174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:37619 every 8 connection(s)
I20260811 11:50:48.369645 16963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:50:48.372895 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.358386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:49969 (local address 127.3.221.126:46025)
0811 11:50:48.358513 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:48.358518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:48.359624 (+ 1106us) server_negotiation.cc:407] Connection header received
0811 11:50:48.360690 (+ 1066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:48.360694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:48.360754 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:48.360846 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:48.362622 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.363201 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.364463 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.364786 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.364882 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.368760 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:48.368764 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:48.368767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:48.368801 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:48.370586 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:48.371091 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:48.371095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:48.371097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:48.371185 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:48.371715 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:48.371719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:48.371720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:48.371949 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:48.372050 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:48.372630 (+ 580us) server_negotiation.cc:299] Negotiation successful
0811 11:50:48.372754 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 11:50:48.373903 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.358967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46025 (local address 127.3.221.67:49969)
0811 11:50:48.359475 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:48.359519 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:48.360421 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:48.361092 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:48.361103 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:48.361649 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:48.362399 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.362426 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.363352 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.363355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.364329 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.364341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.364959 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.364964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.365171 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.366621 (+ 1450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:48.366645 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:48.368527 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:48.370707 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:48.370713 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:48.370729 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:48.370989 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:48.371292 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:48.371295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:48.371297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:48.371586 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:48.372067 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:48.372075 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:48.372350 (+ 275us) client_negotiation.cc:769] Sending connection context
0811 11:50:48.372542 (+ 192us) client_negotiation.cc:240] Negotiation successful
0811 11:50:48.372885 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":145,"threads_started":1}
I20260811 11:50:48.375283 17175 heartbeater.cc:347] Connected to a master server at 127.3.221.126:46025
I20260811 11:50:48.375452 17175 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:48.375998 17175 heartbeater.cc:511] Master 127.3.221.126:46025 requested a full tablet report, sending...
I20260811 11:50:48.376416 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 16963
I20260811 11:50:48.376565 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:50:48.376906 3957 external_mini_cluster.cc:1428] Setting key 2e0b1c1b0a974a1426bc6d8eb0024a37
I20260811 11:50:48.377100 16320 ts_manager.cc:194] Registered new tserver with Master: fba11c35b7c549fd85bf67db6558f280 (127.3.221.67:37619)
I20260811 11:50:48.377878 16320 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:49969
I20260811 11:50:48.379192 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:50:48.385540 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.377657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:38589 (local address 127.3.221.126:46025)
0811 11:50:48.377773 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:48.377778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:48.378030 (+ 252us) server_negotiation.cc:407] Connection header received
0811 11:50:48.378213 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:48.378230 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:48.378282 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:48.378332 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:48.378872 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.379537 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.380608 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.380900 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.381007 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.382151 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:48.382155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:48.382158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:48.382186 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:48.383833 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:48.384282 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:48.384286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:48.384288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:48.384376 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:48.384648 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:48.384651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:48.384653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:48.384860 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:48.384952 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:48.385252 (+ 300us) server_negotiation.cc:299] Negotiation successful
0811 11:50:48.385374 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 11:50:48.385562 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.377564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46025 (local address 127.3.221.67:38589)
0811 11:50:48.377947 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:48.377961 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:48.378104 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:48.378474 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:48.378476 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:48.378653 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:48.378758 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.378763 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.379675 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.379679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.380475 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.380486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.381041 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.381044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.381124 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.381649 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:48.381661 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:48.382008 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:48.383954 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:48.383957 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:48.383959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:48.384179 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:48.384466 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:48.384470 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:48.384472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:48.384562 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:48.384957 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:48.384960 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:48.385061 (+ 101us) client_negotiation.cc:769] Sending connection context
0811 11:50:48.385158 (+ 97us) client_negotiation.cc:240] Negotiation successful
0811 11:50:48.385367 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":102,"threads_started":1}
Aug 11 11:50:48 dist-test-slave-bndk krb5kdc[16255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449048, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 11:50:48 dist-test-slave-bndk krb5kdc[16255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:48 dist-test-slave-bndk krb5kdc[16255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
W20260811 11:50:48.407049 16411 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:48.400659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53236 (local address 127.3.221.126:46025)
0811 11:50:48.400782 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:48.400785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:48.400798 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:50:48.400847 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:48.400849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:48.400889 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:48.400960 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:48.401797 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.402344 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.403187 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.403406 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.403497 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.406923 (+ 3426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53236: BlockingRecv error: recv got EOF from 127.0.0.1:53236 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 11 11:50:48 dist-test-slave-bndk krb5kdc[16255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449048, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM
I20260811 11:50:48.416898 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.408621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53240 (local address 127.3.221.126:46025)
0811 11:50:48.408738 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:48.408743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:48.408797 (+ 54us) server_negotiation.cc:407] Connection header received
0811 11:50:48.408954 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:48.408957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:48.409004 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:48.409092 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:48.409779 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.410293 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.411118 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.411352 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.411443 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.413952 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:48.413956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:48.413958 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:48.413987 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:48.415376 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:48.415787 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:48.415789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:48.415790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:48.415864 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:48.416058 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:48.416060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:48.416062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:48.416235 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:48.416366 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:48.416637 (+ 271us) server_negotiation.cc:299] Negotiation successful
0811 11:50:48.416738 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 11:50:48.419945 16320 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:50:48.423081 16320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:50:48.437427 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.431672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:37619 (local address 127.0.0.1:56334)
0811 11:50:48.432199 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:48.432214 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:48.432315 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:48.433551 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:48.433554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:48.433570 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:48.433653 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.433660 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.434610 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.434613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.435875 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.435886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.436840 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.436844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.436954 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.437001 (+ 47us) client_negotiation.cc:769] Sending connection context
0811 11:50:48.437109 (+ 108us) client_negotiation.cc:240] Negotiation successful
0811 11:50:48.437275 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":90,"threads_started":1}
I20260811 11:50:48.437438 17220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.432160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56334 (local address 127.3.221.67:37619)
0811 11:50:48.433078 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:48.433089 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:48.433108 (+ 19us) server_negotiation.cc:407] Connection header received
0811 11:50:48.433174 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:48.433179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:48.433401 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:48.433569 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:48.433757 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.434496 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.435986 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.436719 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.436832 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.437025 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:48.437143 (+ 118us) server_negotiation.cc:299] Negotiation successful
0811 11:50:48.437280 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":825,"thread_start_us":125,"threads_started":1}
I20260811 11:50:48.437882 17222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.432232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:44233 (local address 127.0.0.1:41868)
0811 11:50:48.432756 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:48.432767 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:48.432850 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:48.433349 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:48.433355 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:48.433377 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:48.433712 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.433723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.435050 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.435053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.436151 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.436160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.437407 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.437411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.437498 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.437523 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 11:50:48.437671 (+ 148us) client_negotiation.cc:240] Negotiation successful
0811 11:50:48.437752 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":182,"thread_start_us":77,"threads_started":1}
I20260811 11:50:48.437927 17223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.432422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41868 (local address 127.3.221.66:44233)
0811 11:50:48.432742 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:48.432754 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:48.432867 (+ 113us) server_negotiation.cc:407] Connection header received
0811 11:50:48.432954 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:48.432959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:48.433156 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:48.433542 (+ 386us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:48.433830 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.434925 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.436265 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.437051 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.437148 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.437331 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:48.437703 (+ 372us) server_negotiation.cc:299] Negotiation successful
0811 11:50:48.437791 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":108,"threads_started":1}
I20260811 11:50:48.438483 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.431564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:45263 (local address 127.0.0.1:59490)
0811 11:50:48.431995 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:48.432044 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:48.432676 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:48.433366 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:48.433370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:48.433389 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:48.433543 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.433550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.436021 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.436023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.436812 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.436823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.438110 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.438114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.438206 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.438229 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 11:50:48.438291 (+ 62us) client_negotiation.cc:240] Negotiation successful
0811 11:50:48.438363 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":1,"thread_start_us":66,"threads_started":1}
I20260811 11:50:48.439010 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.431705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59490 (local address 127.3.221.65:45263)
0811 11:50:48.432591 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:48.432598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:48.432624 (+ 26us) server_negotiation.cc:407] Connection header received
0811 11:50:48.432891 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:48.432897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:48.433146 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:48.433366 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:48.433676 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.434921 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.436956 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.437784 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.437886 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.438746 (+ 860us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:48.438814 (+ 68us) server_negotiation.cc:299] Negotiation successful
0811 11:50:48.438909 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":62,"threads_started":1}
I20260811 11:50:48.440976 17072 tablet_service.cc:1467] Processing CreateTablet for tablet d27b78e7d3e548fabcc0cc4d694c792a (DEFAULT_TABLE table=test-table [id=0dd6c38f0c3c41619025cb69428bc1e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:48.441071 16841 tablet_service.cc:1467] Processing CreateTablet for tablet d27b78e7d3e548fabcc0cc4d694c792a (DEFAULT_TABLE table=test-table [id=0dd6c38f0c3c41619025cb69428bc1e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:48.442026 17072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d27b78e7d3e548fabcc0cc4d694c792a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:48.442124 16841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d27b78e7d3e548fabcc0cc4d694c792a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:48.441753 16584 tablet_service.cc:1467] Processing CreateTablet for tablet d27b78e7d3e548fabcc0cc4d694c792a (DEFAULT_TABLE table=test-table [id=0dd6c38f0c3c41619025cb69428bc1e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:48.442682 16584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d27b78e7d3e548fabcc0cc4d694c792a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:48.449118 17224 tablet_bootstrap.cc:492] T d27b78e7d3e548fabcc0cc4d694c792a P fba11c35b7c549fd85bf67db6558f280: Bootstrap starting.
I20260811 11:50:48.450304 17225 tablet_bootstrap.cc:492] T d27b78e7d3e548fabcc0cc4d694c792a P b457496f7b59437eaec5d6efdd22b306: Bootstrap starting.
I20260811 11:50:48.450722 17226 tablet_bootstrap.cc:492] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144: Bootstrap starting.
I20260811 11:50:48.451691 17224 tablet_bootstrap.cc:654] T d27b78e7d3e548fabcc0cc4d694c792a P fba11c35b7c549fd85bf67db6558f280: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:48.452198 17225 tablet_bootstrap.cc:654] T d27b78e7d3e548fabcc0cc4d694c792a P b457496f7b59437eaec5d6efdd22b306: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:48.452646 17224 log.cc:824] T d27b78e7d3e548fabcc0cc4d694c792a P fba11c35b7c549fd85bf67db6558f280: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:48.452867 17226 tablet_bootstrap.cc:654] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:48.452908 17225 log.cc:824] T d27b78e7d3e548fabcc0cc4d694c792a P b457496f7b59437eaec5d6efdd22b306: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:48.453703 17226 log.cc:824] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:48.455199 17225 tablet_bootstrap.cc:492] T d27b78e7d3e548fabcc0cc4d694c792a P b457496f7b59437eaec5d6efdd22b306: No bootstrap required, opened a new log
I20260811 11:50:48.455214 17224 tablet_bootstrap.cc:492] T d27b78e7d3e548fabcc0cc4d694c792a P fba11c35b7c549fd85bf67db6558f280: No bootstrap required, opened a new log
I20260811 11:50:48.455416 17224 ts_tablet_manager.cc:1397] T d27b78e7d3e548fabcc0cc4d694c792a P fba11c35b7c549fd85bf67db6558f280: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 11:50:48.455417 17225 ts_tablet_manager.cc:1397] T d27b78e7d3e548fabcc0cc4d694c792a P b457496f7b59437eaec5d6efdd22b306: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260811 11:50:48.455600 17226 tablet_bootstrap.cc:492] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144: No bootstrap required, opened a new log
I20260811 11:50:48.455782 17226 ts_tablet_manager.cc:1397] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260811 11:50:48.458802 17226 raft_consensus.cc:348] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } }
I20260811 11:50:48.459041 17226 raft_consensus.cc:374] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:48.459115 17226 raft_consensus.cc:729] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df141eeadc64b96ad0d38bd94281144, State: Initialized, Role: FOLLOWER
I20260811 11:50:48.459416 17224 raft_consensus.cc:348] T d27b78e7d3e548fabcc0cc4d694c792a P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } }
I20260811 11:50:48.459565 17226 consensus_queue.cc:260] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } }
I20260811 11:50:48.459677 17224 raft_consensus.cc:374] T d27b78e7d3e548fabcc0cc4d694c792a P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:48.459766 17224 raft_consensus.cc:729] T d27b78e7d3e548fabcc0cc4d694c792a P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fba11c35b7c549fd85bf67db6558f280, State: Initialized, Role: FOLLOWER
I20260811 11:50:48.459914 17225 raft_consensus.cc:348] T d27b78e7d3e548fabcc0cc4d694c792a P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } }
I20260811 11:50:48.460225 17225 raft_consensus.cc:374] T d27b78e7d3e548fabcc0cc4d694c792a P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:48.460209 17224 consensus_queue.cc:260] T d27b78e7d3e548fabcc0cc4d694c792a P fba11c35b7c549fd85bf67db6558f280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } }
I20260811 11:50:48.460326 17225 raft_consensus.cc:729] T d27b78e7d3e548fabcc0cc4d694c792a P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b457496f7b59437eaec5d6efdd22b306, State: Initialized, Role: FOLLOWER
I20260811 11:50:48.460664 17226 ts_tablet_manager.cc:1428] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260811 11:50:48.460860 17225 consensus_queue.cc:260] T d27b78e7d3e548fabcc0cc4d694c792a P b457496f7b59437eaec5d6efdd22b306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } }
I20260811 11:50:48.461203 17224 ts_tablet_manager.cc:1428] T d27b78e7d3e548fabcc0cc4d694c792a P fba11c35b7c549fd85bf67db6558f280: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260811 11:50:48.461201 16950 heartbeater.cc:503] Master 127.3.221.126:46025 was elected leader, sending a full tablet report...
I20260811 11:50:48.461603 17175 heartbeater.cc:503] Master 127.3.221.126:46025 was elected leader, sending a full tablet report...
I20260811 11:50:48.462715 16728 heartbeater.cc:503] Master 127.3.221.126:46025 was elected leader, sending a full tablet report...
I20260811 11:50:48.462903 17225 ts_tablet_manager.cc:1428] T d27b78e7d3e548fabcc0cc4d694c792a P b457496f7b59437eaec5d6efdd22b306: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260811 11:50:48.576182 17230 raft_consensus.cc:482] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:50:48.576416 17230 raft_consensus.cc:504] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } }
I20260811 11:50:48.577639 17230 leader_election.cc:288] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b457496f7b59437eaec5d6efdd22b306 (127.3.221.65:45263), fba11c35b7c549fd85bf67db6558f280 (127.3.221.67:37619)
I20260811 11:50:48.581710 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.577968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:37619 (local address 127.3.221.66:49991)
0811 11:50:48.578174 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:48.578190 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:48.578307 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:48.578548 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:48.578551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:48.578574 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:48.578699 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.578705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.579472 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.579475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.580520 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.580532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.581277 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.581280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.581387 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.581414 (+ 27us) client_negotiation.cc:769] Sending connection context
0811 11:50:48.581485 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 11:50:48.581565 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"spinlock_wait_cycles":3328}
I20260811 11:50:48.581710 17220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.578193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:49991 (local address 127.3.221.67:37619)
0811 11:50:48.578318 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:48.578323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:48.578342 (+ 19us) server_negotiation.cc:407] Connection header received
0811 11:50:48.578398 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:48.578400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:48.578441 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:48.578508 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:48.578824 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.579364 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.580660 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.581131 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.581281 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.581341 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:48.581526 (+ 185us) server_negotiation.cc:299] Negotiation successful
0811 11:50:48.581607 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 11:50:48.582330 17233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.578007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:45263 (local address 127.3.221.66:36197)
0811 11:50:48.578458 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:48.578468 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:48.578565 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:48.578788 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:48.578791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:48.578809 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:48.579248 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.579254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.579997 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.580001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.580928 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.580937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.581921 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.581925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.582021 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.582043 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 11:50:48.582111 (+ 68us) client_negotiation.cc:240] Negotiation successful
0811 11:50:48.582172 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":158,"threads_started":1}
I20260811 11:50:48.582381 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.578115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:36197 (local address 127.3.221.65:45263)
0811 11:50:48.578249 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:48.578254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:48.578562 (+ 308us) server_negotiation.cc:407] Connection header received
0811 11:50:48.578646 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:48.578648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:48.578689 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:48.578747 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:48.579372 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.579879 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.581079 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.581673 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.581799 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.581851 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:48.582137 (+ 286us) server_negotiation.cc:299] Negotiation successful
0811 11:50:48.582222 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 11:50:48.582427 17100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d27b78e7d3e548fabcc0cc4d694c792a" candidate_uuid: "4df141eeadc64b96ad0d38bd94281144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fba11c35b7c549fd85bf67db6558f280" is_pre_election: true
I20260811 11:50:48.582849 17100 raft_consensus.cc:2454] T d27b78e7d3e548fabcc0cc4d694c792a P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df141eeadc64b96ad0d38bd94281144 in term 0.
I20260811 11:50:48.583165 16625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d27b78e7d3e548fabcc0cc4d694c792a" candidate_uuid: "4df141eeadc64b96ad0d38bd94281144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b457496f7b59437eaec5d6efdd22b306" is_pre_election: true
I20260811 11:50:48.583410 16774 leader_election.cc:302] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df141eeadc64b96ad0d38bd94281144, fba11c35b7c549fd85bf67db6558f280; no voters:
I20260811 11:50:48.583499 16625 raft_consensus.cc:2454] T d27b78e7d3e548fabcc0cc4d694c792a P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df141eeadc64b96ad0d38bd94281144 in term 0.
I20260811 11:50:48.583696 17230 raft_consensus.cc:2781] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:50:48.583809 17230 raft_consensus.cc:482] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:50:48.583868 17230 raft_consensus.cc:3037] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:48.584929 17230 raft_consensus.cc:504] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } }
I20260811 11:50:48.585325 17230 leader_election.cc:288] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144 [CANDIDATE]: Term 1 election: Requested vote from peers b457496f7b59437eaec5d6efdd22b306 (127.3.221.65:45263), fba11c35b7c549fd85bf67db6558f280 (127.3.221.67:37619)
I20260811 11:50:48.585700 17100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d27b78e7d3e548fabcc0cc4d694c792a" candidate_uuid: "4df141eeadc64b96ad0d38bd94281144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fba11c35b7c549fd85bf67db6558f280"
I20260811 11:50:48.585719 16625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d27b78e7d3e548fabcc0cc4d694c792a" candidate_uuid: "4df141eeadc64b96ad0d38bd94281144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b457496f7b59437eaec5d6efdd22b306"
I20260811 11:50:48.585830 17100 raft_consensus.cc:3037] T d27b78e7d3e548fabcc0cc4d694c792a P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:48.585837 16625 raft_consensus.cc:3037] T d27b78e7d3e548fabcc0cc4d694c792a P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:48.586977 16625 raft_consensus.cc:2454] T d27b78e7d3e548fabcc0cc4d694c792a P b457496f7b59437eaec5d6efdd22b306 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df141eeadc64b96ad0d38bd94281144 in term 1.
I20260811 11:50:48.586992 17100 raft_consensus.cc:2454] T d27b78e7d3e548fabcc0cc4d694c792a P fba11c35b7c549fd85bf67db6558f280 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df141eeadc64b96ad0d38bd94281144 in term 1.
I20260811 11:50:48.587320 16768 leader_election.cc:302] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df141eeadc64b96ad0d38bd94281144, b457496f7b59437eaec5d6efdd22b306; no voters:
I20260811 11:50:48.587481 17230 raft_consensus.cc:2781] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:48.587756 17230 raft_consensus.cc:686] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144 [term 1 LEADER]: Becoming Leader. State: Replica: 4df141eeadc64b96ad0d38bd94281144, State: Running, Role: LEADER
I20260811 11:50:48.588099 17230 consensus_queue.cc:237] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } }
I20260811 11:50:48.591413 16313 catalog_manager.cc:5495] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4df141eeadc64b96ad0d38bd94281144 (127.3.221.66). New cstate: current_term: 1 leader_uuid: "4df141eeadc64b96ad0d38bd94281144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } health_report { overall_health: HEALTHY } } }
W20260811 11:50:48.621088 17178 tablet.cc:2369] T d27b78e7d3e548fabcc0cc4d694c792a P fba11c35b7c549fd85bf67db6558f280: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:50:48.628172 17223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.624644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41878 (local address 127.3.221.66:44233)
0811 11:50:48.624791 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:48.624795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:48.624811 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:50:48.624965 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:48.624968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:48.625027 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:48.625118 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:48.625390 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.625891 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.626719 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.626947 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.627036 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.627301 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:48.627782 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:48.627882 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:48.627965 (+ 83us) server_negotiation.cc:299] Negotiation successful
0811 11:50:48.628024 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 11:50:48.634032 17100 raft_consensus.cc:1261] T d27b78e7d3e548fabcc0cc4d694c792a P fba11c35b7c549fd85bf67db6558f280 [term 1 FOLLOWER]: Refusing update from remote peer 4df141eeadc64b96ad0d38bd94281144: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:48.634037 16625 raft_consensus.cc:1261] T d27b78e7d3e548fabcc0cc4d694c792a P b457496f7b59437eaec5d6efdd22b306 [term 1 FOLLOWER]: Refusing update from remote peer 4df141eeadc64b96ad0d38bd94281144: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:48.634723 17230 consensus_queue.cc:1035] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:48.634948 17237 consensus_queue.cc:1035] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:48.642761 17241 mvcc.cc:205] Tried to move back new op lower bound from 7317295303197650944 to 7317295303019618304. Current Snapshot: MvccSnapshot[applied={T|T < 7317295303197650944}]
I20260811 11:50:48.644694 17245 mvcc.cc:205] Tried to move back new op lower bound from 7317295303197650944 to 7317295303019618304. Current Snapshot: MvccSnapshot[applied={T|T < 7317295303197650944}]
I20260811 11:50:48.654757 16313 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:48.654965 16313 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:48.657727 16313 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9: Sending DeleteTablet for 3 replicas of tablet d27b78e7d3e548fabcc0cc4d694c792a
I20260811 11:50:48.658699 16584 tablet_service.cc:1514] Processing DeleteTablet for tablet d27b78e7d3e548fabcc0cc4d694c792a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:48 UTC) from {username='oryx', principal='oryx/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:59490
I20260811 11:50:48.658707 17072 tablet_service.cc:1514] Processing DeleteTablet for tablet d27b78e7d3e548fabcc0cc4d694c792a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:48 UTC) from {username='oryx', principal='oryx/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:56334
I20260811 11:50:48.659248 16841 tablet_service.cc:1514] Processing DeleteTablet for tablet d27b78e7d3e548fabcc0cc4d694c792a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:48 UTC) from {username='oryx', principal='oryx/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:41868
I20260811 11:50:48.659286 17264 tablet_replica.cc:330] T d27b78e7d3e548fabcc0cc4d694c792a P fba11c35b7c549fd85bf67db6558f280: stopping tablet replica
I20260811 11:50:48.659255 16313 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260811 11:50:48.659668 17267 tablet_replica.cc:330] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144: stopping tablet replica
W20260811 11:50:48.659714 16313 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:50:48.659968 17267 raft_consensus.cc:2229] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:50:48.659982 17264 raft_consensus.cc:2229] T d27b78e7d3e548fabcc0cc4d694c792a P fba11c35b7c549fd85bf67db6558f280 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:48.660399 17267 raft_consensus.cc:2258] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:48.660516 17264 raft_consensus.cc:2258] T d27b78e7d3e548fabcc0cc4d694c792a P fba11c35b7c549fd85bf67db6558f280 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:48.662317 17267 ts_tablet_manager.cc:1905] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:48.662328 17264 ts_tablet_manager.cc:1905] T d27b78e7d3e548fabcc0cc4d694c792a P fba11c35b7c549fd85bf67db6558f280: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:48.665014 16841 tablet_service.cc:1467] Processing CreateTablet for tablet f8ffc7146c364572aac283619f5acbed (DEFAULT_TABLE table=test-table [id=b219fa806411408c8888a565c66b6b91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:48.661099 17262 tablet_replica.cc:330] T d27b78e7d3e548fabcc0cc4d694c792a P b457496f7b59437eaec5d6efdd22b306: stopping tablet replica
I20260811 11:50:48.665202 17267 ts_tablet_manager.cc:1918] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:50:48.665331 16841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ffc7146c364572aac283619f5acbed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:48.665704 17262 raft_consensus.cc:2229] T d27b78e7d3e548fabcc0cc4d694c792a P b457496f7b59437eaec5d6efdd22b306 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:48.665730 17072 tablet_service.cc:1467] Processing CreateTablet for tablet f8ffc7146c364572aac283619f5acbed (DEFAULT_TABLE table=test-table [id=b219fa806411408c8888a565c66b6b91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:48.665980 17072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ffc7146c364572aac283619f5acbed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:48.666085 17264 ts_tablet_manager.cc:1918] T d27b78e7d3e548fabcc0cc4d694c792a P fba11c35b7c549fd85bf67db6558f280: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:50:48.666220 17264 log.cc:1196] T d27b78e7d3e548fabcc0cc4d694c792a P fba11c35b7c549fd85bf67db6558f280: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/d27b78e7d3e548fabcc0cc4d694c792a
I20260811 11:50:48.666623 17262 raft_consensus.cc:2258] T d27b78e7d3e548fabcc0cc4d694c792a P b457496f7b59437eaec5d6efdd22b306 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:48.667049 17264 ts_tablet_manager.cc:1939] T d27b78e7d3e548fabcc0cc4d694c792a P fba11c35b7c549fd85bf67db6558f280: Deleting consensus metadata
I20260811 11:50:48.665321 17267 log.cc:1196] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/d27b78e7d3e548fabcc0cc4d694c792a
I20260811 11:50:48.668090 17226 tablet_bootstrap.cc:492] T f8ffc7146c364572aac283619f5acbed P 4df141eeadc64b96ad0d38bd94281144: Bootstrap starting.
I20260811 11:50:48.668242 17267 ts_tablet_manager.cc:1939] T d27b78e7d3e548fabcc0cc4d694c792a P 4df141eeadc64b96ad0d38bd94281144: Deleting consensus metadata
I20260811 11:50:48.669060 17226 tablet_bootstrap.cc:654] T f8ffc7146c364572aac283619f5acbed P 4df141eeadc64b96ad0d38bd94281144: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:48.669426 16300 catalog_manager.cc:4841] TS 4df141eeadc64b96ad0d38bd94281144 (127.3.221.66:44233): tablet d27b78e7d3e548fabcc0cc4d694c792a (table test-table [id=0dd6c38f0c3c41619025cb69428bc1e6]) successfully deleted
I20260811 11:50:48.669696 17224 tablet_bootstrap.cc:492] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280: Bootstrap starting.
I20260811 11:50:48.670082 17262 ts_tablet_manager.cc:1905] T d27b78e7d3e548fabcc0cc4d694c792a P b457496f7b59437eaec5d6efdd22b306: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:48.670630 17224 tablet_bootstrap.cc:654] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:48.671458 16300 catalog_manager.cc:4841] TS fba11c35b7c549fd85bf67db6558f280 (127.3.221.67:37619): tablet d27b78e7d3e548fabcc0cc4d694c792a (table test-table [id=0dd6c38f0c3c41619025cb69428bc1e6]) successfully deleted
I20260811 11:50:48.672068 16584 tablet_service.cc:1467] Processing CreateTablet for tablet f8ffc7146c364572aac283619f5acbed (DEFAULT_TABLE table=test-table [id=b219fa806411408c8888a565c66b6b91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:48.672302 17224 tablet_bootstrap.cc:492] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280: No bootstrap required, opened a new log
I20260811 11:50:48.672318 16584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ffc7146c364572aac283619f5acbed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:48.672376 17224 ts_tablet_manager.cc:1397] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260811 11:50:48.672571 17262 ts_tablet_manager.cc:1918] T d27b78e7d3e548fabcc0cc4d694c792a P b457496f7b59437eaec5d6efdd22b306: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:50:48.672657 17262 log.cc:1196] T d27b78e7d3e548fabcc0cc4d694c792a P b457496f7b59437eaec5d6efdd22b306: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-0/wal/wals/d27b78e7d3e548fabcc0cc4d694c792a
I20260811 11:50:48.672791 17224 raft_consensus.cc:348] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:48.672909 17224 raft_consensus.cc:374] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:48.672935 17262 ts_tablet_manager.cc:1939] T d27b78e7d3e548fabcc0cc4d694c792a P b457496f7b59437eaec5d6efdd22b306: Deleting consensus metadata
I20260811 11:50:48.672937 17224 raft_consensus.cc:729] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fba11c35b7c549fd85bf67db6558f280, State: Initialized, Role: FOLLOWER
I20260811 11:50:48.673054 17224 consensus_queue.cc:260] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:48.674403 17225 tablet_bootstrap.cc:492] T f8ffc7146c364572aac283619f5acbed P b457496f7b59437eaec5d6efdd22b306: Bootstrap starting.
I20260811 11:50:48.674890 16297 catalog_manager.cc:4841] TS b457496f7b59437eaec5d6efdd22b306 (127.3.221.65:45263): tablet d27b78e7d3e548fabcc0cc4d694c792a (table test-table [id=0dd6c38f0c3c41619025cb69428bc1e6]) successfully deleted
I20260811 11:50:48.675238 17224 ts_tablet_manager.cc:1428] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260811 11:50:48.675274 17225 tablet_bootstrap.cc:654] T f8ffc7146c364572aac283619f5acbed P b457496f7b59437eaec5d6efdd22b306: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:48.677752 17226 tablet_bootstrap.cc:492] T f8ffc7146c364572aac283619f5acbed P 4df141eeadc64b96ad0d38bd94281144: No bootstrap required, opened a new log
I20260811 11:50:48.677883 17226 ts_tablet_manager.cc:1397] T f8ffc7146c364572aac283619f5acbed P 4df141eeadc64b96ad0d38bd94281144: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20260811 11:50:48.677923 17225 tablet_bootstrap.cc:492] T f8ffc7146c364572aac283619f5acbed P b457496f7b59437eaec5d6efdd22b306: No bootstrap required, opened a new log
I20260811 11:50:48.677994 17225 ts_tablet_manager.cc:1397] T f8ffc7146c364572aac283619f5acbed P b457496f7b59437eaec5d6efdd22b306: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260811 11:50:48.678395 17226 raft_consensus.cc:348] T f8ffc7146c364572aac283619f5acbed P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:48.678449 17225 raft_consensus.cc:348] T f8ffc7146c364572aac283619f5acbed P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:48.678515 17226 raft_consensus.cc:374] T f8ffc7146c364572aac283619f5acbed P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:48.678537 17225 raft_consensus.cc:374] T f8ffc7146c364572aac283619f5acbed P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:48.678546 17226 raft_consensus.cc:729] T f8ffc7146c364572aac283619f5acbed P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df141eeadc64b96ad0d38bd94281144, State: Initialized, Role: FOLLOWER
I20260811 11:50:48.678568 17225 raft_consensus.cc:729] T f8ffc7146c364572aac283619f5acbed P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b457496f7b59437eaec5d6efdd22b306, State: Initialized, Role: FOLLOWER
I20260811 11:50:48.678675 17225 consensus_queue.cc:260] T f8ffc7146c364572aac283619f5acbed P b457496f7b59437eaec5d6efdd22b306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:48.678678 17226 consensus_queue.cc:260] T f8ffc7146c364572aac283619f5acbed P 4df141eeadc64b96ad0d38bd94281144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:48.678933 17225 ts_tablet_manager.cc:1428] T f8ffc7146c364572aac283619f5acbed P b457496f7b59437eaec5d6efdd22b306: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260811 11:50:48.678932 17226 ts_tablet_manager.cc:1428] T f8ffc7146c364572aac283619f5acbed P 4df141eeadc64b96ad0d38bd94281144: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 11:50:48.774991 17231 raft_consensus.cc:482] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:50:48.775188 17231 raft_consensus.cc:504] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:48.776635 17231 leader_election.cc:288] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b457496f7b59437eaec5d6efdd22b306 (127.3.221.65:45263), 4df141eeadc64b96ad0d38bd94281144 (127.3.221.66:44233)
I20260811 11:50:48.781205 17223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.777583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:47465 (local address 127.3.221.66:44233)
0811 11:50:48.777710 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:48.777715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:48.777728 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:50:48.777778 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:48.777781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:48.777840 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:48.777902 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:48.778192 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.778950 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.780258 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.780668 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.780783 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.780842 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:48.781017 (+ 175us) server_negotiation.cc:299] Negotiation successful
0811 11:50:48.781086 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 11:50:48.781208 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.777030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:44233 (local address 127.3.221.67:47465)
0811 11:50:48.777313 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:48.777334 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:48.777441 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:48.777935 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:48.777938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:48.777957 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:48.778064 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.778070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.779080 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.779083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.780146 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.780154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.780782 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.780786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.780895 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.780918 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 11:50:48.780985 (+ 67us) client_negotiation.cc:240] Negotiation successful
0811 11:50:48.781063 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":2,"spinlock_wait_cycles":9600}
I20260811 11:50:48.782073 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.777048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:45263 (local address 127.3.221.67:53335)
0811 11:50:48.777313 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:48.777329 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:48.777441 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:48.777765 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:48.777768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:48.777809 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:48.777955 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.777964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.778725 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.778728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.779959 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.779970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.781672 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.781675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.781753 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.781772 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 11:50:48.781836 (+ 64us) client_negotiation.cc:240] Negotiation successful
0811 11:50:48.781905 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":25,"spinlock_wait_cycles":22144,"thread_start_us":111,"threads_started":1}
I20260811 11:50:48.782079 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.777143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:53335 (local address 127.3.221.65:45263)
0811 11:50:48.777411 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:48.777415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:48.777433 (+ 18us) server_negotiation.cc:407] Connection header received
0811 11:50:48.777589 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:48.777593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:48.777643 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:48.777711 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:48.778076 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.778617 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.780117 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.781539 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.781674 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.781733 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:48.781890 (+ 157us) server_negotiation.cc:299] Negotiation successful
0811 11:50:48.781972 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260811 11:50:48.782202 16875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ffc7146c364572aac283619f5acbed" candidate_uuid: "fba11c35b7c549fd85bf67db6558f280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df141eeadc64b96ad0d38bd94281144" is_pre_election: true
I20260811 11:50:48.782485 16875 raft_consensus.cc:2454] T f8ffc7146c364572aac283619f5acbed P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fba11c35b7c549fd85bf67db6558f280 in term 0.
I20260811 11:50:48.782953 16993 leader_election.cc:302] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df141eeadc64b96ad0d38bd94281144, fba11c35b7c549fd85bf67db6558f280; no voters:
I20260811 11:50:48.783207 17231 raft_consensus.cc:2781] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:50:48.783267 17231 raft_consensus.cc:482] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:50:48.783300 17231 raft_consensus.cc:3037] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:48.783370 16625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ffc7146c364572aac283619f5acbed" candidate_uuid: "fba11c35b7c549fd85bf67db6558f280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b457496f7b59437eaec5d6efdd22b306" is_pre_election: true
I20260811 11:50:48.783519 16625 raft_consensus.cc:2454] T f8ffc7146c364572aac283619f5acbed P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fba11c35b7c549fd85bf67db6558f280 in term 0.
I20260811 11:50:48.784525 17231 raft_consensus.cc:504] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:48.784891 17231 leader_election.cc:288] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280 [CANDIDATE]: Term 1 election: Requested vote from peers b457496f7b59437eaec5d6efdd22b306 (127.3.221.65:45263), 4df141eeadc64b96ad0d38bd94281144 (127.3.221.66:44233)
I20260811 11:50:48.785255 16625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ffc7146c364572aac283619f5acbed" candidate_uuid: "fba11c35b7c549fd85bf67db6558f280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b457496f7b59437eaec5d6efdd22b306"
I20260811 11:50:48.785383 16625 raft_consensus.cc:3037] T f8ffc7146c364572aac283619f5acbed P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:48.786527 16625 raft_consensus.cc:2454] T f8ffc7146c364572aac283619f5acbed P b457496f7b59437eaec5d6efdd22b306 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fba11c35b7c549fd85bf67db6558f280 in term 1.
I20260811 11:50:48.786793 16875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ffc7146c364572aac283619f5acbed" candidate_uuid: "fba11c35b7c549fd85bf67db6558f280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df141eeadc64b96ad0d38bd94281144"
I20260811 11:50:48.786924 16875 raft_consensus.cc:3037] T f8ffc7146c364572aac283619f5acbed P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:48.788075 16875 raft_consensus.cc:2454] T f8ffc7146c364572aac283619f5acbed P 4df141eeadc64b96ad0d38bd94281144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fba11c35b7c549fd85bf67db6558f280 in term 1.
I20260811 11:50:48.788475 16990 leader_election.cc:302] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b457496f7b59437eaec5d6efdd22b306, fba11c35b7c549fd85bf67db6558f280; no voters:
I20260811 11:50:48.788677 17231 raft_consensus.cc:2781] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:48.788929 17231 raft_consensus.cc:686] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280 [term 1 LEADER]: Becoming Leader. State: Replica: fba11c35b7c549fd85bf67db6558f280, State: Running, Role: LEADER
I20260811 11:50:48.789206 17231 consensus_queue.cc:237] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:48.791764 16313 catalog_manager.cc:5495] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280 reported cstate change: term changed from 0 to 1, leader changed from <none> to fba11c35b7c549fd85bf67db6558f280 (127.3.221.67). New cstate: current_term: 1 leader_uuid: "fba11c35b7c549fd85bf67db6558f280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } health_report { overall_health: HEALTHY } } }
Aug 11 11:50:48 dist-test-slave-bndk krb5kdc[16255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449047, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.126@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM
I20260811 11:50:48.855439 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.847155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53244 (local address 127.3.221.126:46025)
0811 11:50:48.847328 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:48.847332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:48.847476 (+ 144us) server_negotiation.cc:407] Connection header received
0811 11:50:48.847596 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:48.847598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:48.847642 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:48.847735 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:48.848479 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.849016 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.849783 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.849991 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.850066 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.852337 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:48.852341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:48.852343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:48.852367 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:48.853709 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:48.854150 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:48.854153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:48.854155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:48.854231 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:48.854555 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:48.854557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:48.854558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:48.854724 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:48.854806 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:48.855161 (+ 355us) server_negotiation.cc:299] Negotiation successful
0811 11:50:48.855323 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":69}
I20260811 11:50:48.855448 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:48.847075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:46025 (local address 127.0.0.1:53244)
0811 11:50:48.847380 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:48.847396 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:48.847521 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:48.847754 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:48.847756 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:48.848254 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:48.848361 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.848368 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.849111 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.849113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.849692 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:48.849699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:48.850077 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:48.850079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:48.850169 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:48.850691 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:48.850702 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:48.852201 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:48.853812 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:48.853820 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:48.853826 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:48.854050 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:48.854372 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:48.854376 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:48.854378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:48.854477 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:48.854820 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:48.854827 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:48.854960 (+ 133us) client_negotiation.cc:769] Sending connection context
0811 11:50:48.855062 (+ 102us) client_negotiation.cc:240] Negotiation successful
0811 11:50:48.855208 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":17,"thread_start_us":126,"threads_started":1}
I20260811 11:50:48.859618 16313 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:53244:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 11:50:48.860347 16313 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:50:48.865401 16841 tablet_service.cc:1467] Processing CreateTablet for tablet c91168539fc64074a9470a9879da8b6c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16f147222a0242bc8b9e0a7be14a7f9b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 11:50:48.865408 16584 tablet_service.cc:1467] Processing CreateTablet for tablet c91168539fc64074a9470a9879da8b6c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16f147222a0242bc8b9e0a7be14a7f9b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 11:50:48.865407 17072 tablet_service.cc:1467] Processing CreateTablet for tablet c91168539fc64074a9470a9879da8b6c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16f147222a0242bc8b9e0a7be14a7f9b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 11:50:48.865844 16841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91168539fc64074a9470a9879da8b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:48.865844 16584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91168539fc64074a9470a9879da8b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:48.865868 17072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91168539fc64074a9470a9879da8b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:48.868464 17224 tablet_bootstrap.cc:492] T c91168539fc64074a9470a9879da8b6c P fba11c35b7c549fd85bf67db6558f280: Bootstrap starting.
I20260811 11:50:48.868561 17226 tablet_bootstrap.cc:492] T c91168539fc64074a9470a9879da8b6c P 4df141eeadc64b96ad0d38bd94281144: Bootstrap starting.
I20260811 11:50:48.868605 17225 tablet_bootstrap.cc:492] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306: Bootstrap starting.
I20260811 11:50:48.869464 17225 tablet_bootstrap.cc:654] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:48.869458 17226 tablet_bootstrap.cc:654] T c91168539fc64074a9470a9879da8b6c P 4df141eeadc64b96ad0d38bd94281144: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:48.869603 17224 tablet_bootstrap.cc:654] T c91168539fc64074a9470a9879da8b6c P fba11c35b7c549fd85bf67db6558f280: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:48.870635 17226 tablet_bootstrap.cc:492] T c91168539fc64074a9470a9879da8b6c P 4df141eeadc64b96ad0d38bd94281144: No bootstrap required, opened a new log
I20260811 11:50:48.870728 17226 ts_tablet_manager.cc:1397] T c91168539fc64074a9470a9879da8b6c P 4df141eeadc64b96ad0d38bd94281144: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:50:48.870788 17224 tablet_bootstrap.cc:492] T c91168539fc64074a9470a9879da8b6c P fba11c35b7c549fd85bf67db6558f280: No bootstrap required, opened a new log
I20260811 11:50:48.870857 17224 ts_tablet_manager.cc:1397] T c91168539fc64074a9470a9879da8b6c P fba11c35b7c549fd85bf67db6558f280: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 11:50:48.870877 17225 tablet_bootstrap.cc:492] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306: No bootstrap required, opened a new log
I20260811 11:50:48.870937 17225 ts_tablet_manager.cc:1397] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:50:48.871157 17226 raft_consensus.cc:348] T c91168539fc64074a9470a9879da8b6c P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:48.871297 17226 raft_consensus.cc:374] T c91168539fc64074a9470a9879da8b6c P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:48.871258 17224 raft_consensus.cc:348] T c91168539fc64074a9470a9879da8b6c P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:48.871331 17226 raft_consensus.cc:729] T c91168539fc64074a9470a9879da8b6c P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df141eeadc64b96ad0d38bd94281144, State: Initialized, Role: FOLLOWER
I20260811 11:50:48.871284 17225 raft_consensus.cc:348] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:48.871366 17225 raft_consensus.cc:374] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:48.871364 17224 raft_consensus.cc:374] T c91168539fc64074a9470a9879da8b6c P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:48.871388 17225 raft_consensus.cc:729] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b457496f7b59437eaec5d6efdd22b306, State: Initialized, Role: FOLLOWER
I20260811 11:50:48.871395 17224 raft_consensus.cc:729] T c91168539fc64074a9470a9879da8b6c P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fba11c35b7c549fd85bf67db6558f280, State: Initialized, Role: FOLLOWER
I20260811 11:50:48.871464 17226 consensus_queue.cc:260] T c91168539fc64074a9470a9879da8b6c P 4df141eeadc64b96ad0d38bd94281144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:48.871492 17225 consensus_queue.cc:260] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:48.871507 17224 consensus_queue.cc:260] T c91168539fc64074a9470a9879da8b6c P fba11c35b7c549fd85bf67db6558f280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:48.871752 17226 ts_tablet_manager.cc:1428] T c91168539fc64074a9470a9879da8b6c P 4df141eeadc64b96ad0d38bd94281144: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:48.871760 17225 ts_tablet_manager.cc:1428] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:48.871713 17230 tablet_replica.cc:439] T c91168539fc64074a9470a9879da8b6c P 4df141eeadc64b96ad0d38bd94281144: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } } }
I20260811 11:50:48.871851 17230 tablet_replica.cc:442] T c91168539fc64074a9470a9879da8b6c P 4df141eeadc64b96ad0d38bd94281144: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:50:48.871963 17232 tablet_replica.cc:439] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } } }
I20260811 11:50:48.872076 17232 tablet_replica.cc:442] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:50:48.872236 17231 tablet_replica.cc:439] T c91168539fc64074a9470a9879da8b6c P fba11c35b7c549fd85bf67db6558f280: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } } }
I20260811 11:50:48.872365 17231 tablet_replica.cc:442] T c91168539fc64074a9470a9879da8b6c P fba11c35b7c549fd85bf67db6558f280: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:50:48.871758 17224 ts_tablet_manager.cc:1428] T c91168539fc64074a9470a9879da8b6c P fba11c35b7c549fd85bf67db6558f280: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260811 11:50:48.894425 16951 tablet.cc:2369] T c91168539fc64074a9470a9879da8b6c P 4df141eeadc64b96ad0d38bd94281144: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:50:48.910272 16729 tablet.cc:2369] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:50:49.170475 17232 raft_consensus.cc:482] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:50:49.170648 17232 raft_consensus.cc:504] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:49.171811 17232 leader_election.cc:288] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4df141eeadc64b96ad0d38bd94281144 (127.3.221.66:44233), fba11c35b7c549fd85bf67db6558f280 (127.3.221.67:37619)
I20260811 11:50:49.176882 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:49.172584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:60127 (local address 127.3.221.67:37619)
0811 11:50:49.172826 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:49.172830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:49.172844 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:50:49.172894 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:49.172896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:49.172934 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:49.173007 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:49.173194 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.173963 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.175536 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.176188 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.176346 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:49.176391 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:49.176661 (+ 270us) server_negotiation.cc:299] Negotiation successful
0811 11:50:49.176740 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260811 11:50:49.176880 17223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:49.172175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:52007 (local address 127.3.221.66:44233)
0811 11:50:49.172322 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:49.172327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:49.172437 (+ 110us) server_negotiation.cc:407] Connection header received
0811 11:50:49.172625 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:49.172630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:49.172686 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:49.172779 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:50:49.173043 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.173864 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.175535 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.176187 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.176319 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:49.176378 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:49.176661 (+ 283us) server_negotiation.cc:299] Negotiation successful
0811 11:50:49.176753 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 11:50:49.176919 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:49.172053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:44233 (local address 127.3.221.65:52007)
0811 11:50:49.172343 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:49.172359 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:49.172492 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:49.172809 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:49.172812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:49.172827 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:49.172938 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:49.172943 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.173970 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.173973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:49.175402 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:49.175415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.176314 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.176319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:49.176489 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:49.176558 (+ 69us) client_negotiation.cc:769] Sending connection context
0811 11:50:49.176636 (+ 78us) client_negotiation.cc:240] Negotiation successful
0811 11:50:49.176740 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":95,"thread_start_us":89,"threads_started":1}
I20260811 11:50:49.176923 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:49.172448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:37619 (local address 127.3.221.65:60127)
0811 11:50:49.172715 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:49.172727 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:49.172808 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:49.173027 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:49.173030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:49.173047 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:50:49.173110 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:49.173114 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.174079 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.174083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:49.175402 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:49.175415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.176344 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.176347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:49.176514 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:49.176561 (+ 47us) client_negotiation.cc:769] Sending connection context
0811 11:50:49.176636 (+ 75us) client_negotiation.cc:240] Negotiation successful
0811 11:50:49.176744 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":132,"thread_start_us":59,"threads_started":1}
I20260811 11:50:49.177461 16875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91168539fc64074a9470a9879da8b6c" candidate_uuid: "b457496f7b59437eaec5d6efdd22b306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df141eeadc64b96ad0d38bd94281144" is_pre_election: true
I20260811 11:50:49.177532 17100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91168539fc64074a9470a9879da8b6c" candidate_uuid: "b457496f7b59437eaec5d6efdd22b306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fba11c35b7c549fd85bf67db6558f280" is_pre_election: true
I20260811 11:50:49.177659 17100 raft_consensus.cc:2454] T c91168539fc64074a9470a9879da8b6c P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b457496f7b59437eaec5d6efdd22b306 in term 0.
I20260811 11:50:49.177660 16875 raft_consensus.cc:2454] T c91168539fc64074a9470a9879da8b6c P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b457496f7b59437eaec5d6efdd22b306 in term 0.
I20260811 11:50:49.178023 16480 leader_election.cc:302] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df141eeadc64b96ad0d38bd94281144, b457496f7b59437eaec5d6efdd22b306; no voters:
I20260811 11:50:49.178258 17232 raft_consensus.cc:2781] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:50:49.178325 17232 raft_consensus.cc:482] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:50:49.178366 17232 raft_consensus.cc:3037] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:49.179386 17232 raft_consensus.cc:504] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:49.179714 17232 leader_election.cc:288] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306 [CANDIDATE]: Term 1 election: Requested vote from peers 4df141eeadc64b96ad0d38bd94281144 (127.3.221.66:44233), fba11c35b7c549fd85bf67db6558f280 (127.3.221.67:37619)
I20260811 11:50:49.180065 16875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91168539fc64074a9470a9879da8b6c" candidate_uuid: "b457496f7b59437eaec5d6efdd22b306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df141eeadc64b96ad0d38bd94281144"
I20260811 11:50:49.180109 17100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91168539fc64074a9470a9879da8b6c" candidate_uuid: "b457496f7b59437eaec5d6efdd22b306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fba11c35b7c549fd85bf67db6558f280"
I20260811 11:50:49.180202 17100 raft_consensus.cc:3037] T c91168539fc64074a9470a9879da8b6c P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:49.180202 16875 raft_consensus.cc:3037] T c91168539fc64074a9470a9879da8b6c P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:49.181337 16875 raft_consensus.cc:2454] T c91168539fc64074a9470a9879da8b6c P 4df141eeadc64b96ad0d38bd94281144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b457496f7b59437eaec5d6efdd22b306 in term 1.
I20260811 11:50:49.181349 17100 raft_consensus.cc:2454] T c91168539fc64074a9470a9879da8b6c P fba11c35b7c549fd85bf67db6558f280 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b457496f7b59437eaec5d6efdd22b306 in term 1.
I20260811 11:50:49.181619 16480 leader_election.cc:302] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b457496f7b59437eaec5d6efdd22b306, fba11c35b7c549fd85bf67db6558f280; no voters:
I20260811 11:50:49.181769 17232 raft_consensus.cc:2781] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:49.181942 17232 raft_consensus.cc:686] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306 [term 1 LEADER]: Becoming Leader. State: Replica: b457496f7b59437eaec5d6efdd22b306, State: Running, Role: LEADER
I20260811 11:50:49.182245 17232 consensus_queue.cc:237] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:49.183405 17297 tablet_replica.cc:439] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306: TxnStatusTablet state changed. Reason: New leader b457496f7b59437eaec5d6efdd22b306. Latest consensus state: current_term: 1 leader_uuid: "b457496f7b59437eaec5d6efdd22b306" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } } }
I20260811 11:50:49.183542 17297 tablet_replica.cc:442] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306: This TxnStatusTablet replica's current role is: LEADER
I20260811 11:50:49.183802 17300 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 11:50:49.183961 17300 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 11:50:49.184842 16318 catalog_manager.cc:5495] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306 reported cstate change: term changed from 0 to 1, leader changed from <none> to b457496f7b59437eaec5d6efdd22b306 (127.3.221.65). New cstate: current_term: 1 leader_uuid: "b457496f7b59437eaec5d6efdd22b306" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } health_report { overall_health: UNKNOWN } } }
I20260811 11:50:49.226603 17231 consensus_queue.cc:1035] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:49.231510 17231 consensus_queue.cc:1035] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:49.239722 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:49.236138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59494 (local address 127.3.221.65:45263)
0811 11:50:49.236258 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:49.236263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:49.236331 (+ 68us) server_negotiation.cc:407] Connection header received
0811 11:50:49.236534 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:49.236536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:49.236604 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:49.236687 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:49.236924 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.237484 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.238300 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.238515 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.238601 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:49.238901 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:49.239328 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:49.239431 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:49.239526 (+ 95us) server_negotiation.cc:299] Negotiation successful
0811 11:50:49.239583 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 11:50:49.239722 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:49.236027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:45263 (local address 127.0.0.1:59494)
0811 11:50:49.236268 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:49.236283 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:49.236399 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:49.236714 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:49.236717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:49.236725 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:50:49.236830 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:49.236835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.237587 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.237591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:49.238192 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:49.238201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.238617 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.238621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:49.238711 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:49.238761 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:50:49.239428 (+ 667us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:50:49.239443 (+ 15us) client_negotiation.cc:769] Sending connection context
0811 11:50:49.239501 (+ 58us) client_negotiation.cc:240] Negotiation successful
0811 11:50:49.239589 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260811 11:50:49.242622 17100 raft_consensus.cc:1261] T c91168539fc64074a9470a9879da8b6c P fba11c35b7c549fd85bf67db6558f280 [term 1 FOLLOWER]: Refusing update from remote peer b457496f7b59437eaec5d6efdd22b306: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:49.242648 16875 raft_consensus.cc:1261] T c91168539fc64074a9470a9879da8b6c P 4df141eeadc64b96ad0d38bd94281144 [term 1 FOLLOWER]: Refusing update from remote peer b457496f7b59437eaec5d6efdd22b306: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:49.243109 17297 consensus_queue.cc:1035] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:49.243368 17232 consensus_queue.cc:1035] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:49.244860 17231 tablet_replica.cc:439] T c91168539fc64074a9470a9879da8b6c P fba11c35b7c549fd85bf67db6558f280: TxnStatusTablet state changed. Reason: New leader b457496f7b59437eaec5d6efdd22b306. Latest consensus state: current_term: 1 leader_uuid: "b457496f7b59437eaec5d6efdd22b306" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } } }
I20260811 11:50:49.245000 17231 tablet_replica.cc:442] T c91168539fc64074a9470a9879da8b6c P fba11c35b7c549fd85bf67db6558f280: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:50:49.244944 17230 tablet_replica.cc:439] T c91168539fc64074a9470a9879da8b6c P 4df141eeadc64b96ad0d38bd94281144: TxnStatusTablet state changed. Reason: New leader b457496f7b59437eaec5d6efdd22b306. Latest consensus state: current_term: 1 leader_uuid: "b457496f7b59437eaec5d6efdd22b306" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } } }
I20260811 11:50:49.245083 17230 tablet_replica.cc:442] T c91168539fc64074a9470a9879da8b6c P 4df141eeadc64b96ad0d38bd94281144: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:50:49.246806 17297 tablet_replica.cc:439] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b457496f7b59437eaec5d6efdd22b306" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } } }
I20260811 11:50:49.246973 17297 tablet_replica.cc:442] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306: This TxnStatusTablet replica's current role is: LEADER
I20260811 11:50:49.247082 17304 mvcc.cc:205] Tried to move back new op lower bound from 7317295305692463104 to 7317295305453101056. Current Snapshot: MvccSnapshot[applied={T|T < 7317295305692463104}]
I20260811 11:50:49.248674 17231 tablet_replica.cc:439] T c91168539fc64074a9470a9879da8b6c P fba11c35b7c549fd85bf67db6558f280: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b457496f7b59437eaec5d6efdd22b306" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } } }
I20260811 11:50:49.248788 17231 tablet_replica.cc:442] T c91168539fc64074a9470a9879da8b6c P fba11c35b7c549fd85bf67db6558f280: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:50:49.250288 17232 tablet_replica.cc:439] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b457496f7b59437eaec5d6efdd22b306" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } } }
I20260811 11:50:49.250406 17232 tablet_replica.cc:442] T c91168539fc64074a9470a9879da8b6c P b457496f7b59437eaec5d6efdd22b306: This TxnStatusTablet replica's current role is: LEADER
I20260811 11:50:49.251700 17230 tablet_replica.cc:439] T c91168539fc64074a9470a9879da8b6c P 4df141eeadc64b96ad0d38bd94281144: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b457496f7b59437eaec5d6efdd22b306" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } } }
I20260811 11:50:49.251813 17230 tablet_replica.cc:442] T c91168539fc64074a9470a9879da8b6c P 4df141eeadc64b96ad0d38bd94281144: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 11:50:49.259042 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:49.254861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56340 (local address 127.3.221.67:37619)
0811 11:50:49.254997 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:49.255002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:49.255093 (+ 91us) server_negotiation.cc:407] Connection header received
0811 11:50:49.255275 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:49.255278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:49.255344 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:49.255455 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:49.255675 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.256483 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.257358 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.257698 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.257795 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:49.258027 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:49.258609 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:49.258720 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:49.258832 (+ 112us) server_negotiation.cc:299] Negotiation successful
0811 11:50:49.258900 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 11:50:49.270040 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:49.266326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:45263 (local address 127.3.221.67:50899)
0811 11:50:49.266704 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:49.266717 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:49.266815 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:49.267049 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:49.267051 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:49.267058 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:50:49.267166 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:49.267170 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.268159 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.268162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:49.268813 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:49.268824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.269395 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.269397 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:49.269465 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:49.269513 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:50:49.269793 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:50:49.269797 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:50:49.269847 (+ 50us) client_negotiation.cc:240] Negotiation successful
0811 11:50:49.269915 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":113,"threads_started":1}
I20260811 11:50:49.270058 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:49.266439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:50899 (local address 127.3.221.65:45263)
0811 11:50:49.266572 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:49.266577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:49.266785 (+ 208us) server_negotiation.cc:407] Connection header received
0811 11:50:49.266901 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:49.266905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:49.266959 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:49.267037 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:49.267251 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.268043 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.268919 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.269284 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.269373 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:49.269632 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:49.269717 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:49.269788 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:49.269859 (+ 71us) server_negotiation.cc:299] Negotiation successful
0811 11:50:49.269926 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 11:50:49.297704 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:49.294008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:54429 (local address 127.3.221.67:37619)
0811 11:50:49.294126 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:49.294131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:49.294446 (+ 315us) server_negotiation.cc:407] Connection header received
0811 11:50:49.294582 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:49.294586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:49.294640 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:49.294708 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:49.294974 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.295643 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.296624 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.296845 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.296930 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:49.297259 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:49.297344 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:49.297404 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:49.297503 (+ 99us) server_negotiation.cc:299] Negotiation successful
0811 11:50:49.297553 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260811 11:50:49.297719 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:49.293875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:37619 (local address 127.3.221.65:54429)
0811 11:50:49.294350 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:49.294366 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:49.294479 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:49.294735 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:49.294738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:49.294747 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:50:49.294876 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:49.294882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.295811 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.295821 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:49.296506 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:49.296517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.296966 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.296970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:49.297063 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:49.297109 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:50:49.297420 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:50:49.297426 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 11:50:49.297482 (+ 56us) client_negotiation.cc:240] Negotiation successful
0811 11:50:49.297564 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":117,"threads_started":1}
I20260811 11:50:49.328282 16318 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:49.328483 16318 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:49.331171 16318 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9: Sending DeleteTablet for 3 replicas of tablet f8ffc7146c364572aac283619f5acbed
I20260811 11:50:49.332285 16841 tablet_service.cc:1514] Processing DeleteTablet for tablet f8ffc7146c364572aac283619f5acbed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:49 UTC) from {username='oryx', principal='oryx/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:41868
I20260811 11:50:49.332754 17339 tablet_replica.cc:330] T f8ffc7146c364572aac283619f5acbed P 4df141eeadc64b96ad0d38bd94281144: stopping tablet replica
I20260811 11:50:49.332834 17072 tablet_service.cc:1514] Processing DeleteTablet for tablet f8ffc7146c364572aac283619f5acbed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:49 UTC) from {username='oryx', principal='oryx/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:56334
I20260811 11:50:49.333079 17339 raft_consensus.cc:2229] T f8ffc7146c364572aac283619f5acbed P 4df141eeadc64b96ad0d38bd94281144 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:49.333299 17339 raft_consensus.cc:2258] T f8ffc7146c364572aac283619f5acbed P 4df141eeadc64b96ad0d38bd94281144 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:49.334144 17339 ts_tablet_manager.cc:1905] T f8ffc7146c364572aac283619f5acbed P 4df141eeadc64b96ad0d38bd94281144: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:49.337530 17339 ts_tablet_manager.cc:1918] T f8ffc7146c364572aac283619f5acbed P 4df141eeadc64b96ad0d38bd94281144: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:50:49.337702 17339 log.cc:1196] T f8ffc7146c364572aac283619f5acbed P 4df141eeadc64b96ad0d38bd94281144: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/f8ffc7146c364572aac283619f5acbed
I20260811 11:50:49.337842 16584 tablet_service.cc:1514] Processing DeleteTablet for tablet f8ffc7146c364572aac283619f5acbed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:49 UTC) from {username='oryx', principal='oryx/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:59490
I20260811 11:50:49.338151 17339 ts_tablet_manager.cc:1939] T f8ffc7146c364572aac283619f5acbed P 4df141eeadc64b96ad0d38bd94281144: Deleting consensus metadata
I20260811 11:50:49.338989 16300 catalog_manager.cc:4841] TS 4df141eeadc64b96ad0d38bd94281144 (127.3.221.66:44233): tablet f8ffc7146c364572aac283619f5acbed (table test-table [id=b219fa806411408c8888a565c66b6b91]) successfully deleted
I20260811 11:50:49.341601 17347 tablet_replica.cc:330] T f8ffc7146c364572aac283619f5acbed P b457496f7b59437eaec5d6efdd22b306: stopping tablet replica
I20260811 11:50:49.341755 17347 raft_consensus.cc:2229] T f8ffc7146c364572aac283619f5acbed P b457496f7b59437eaec5d6efdd22b306 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:49.341877 17347 raft_consensus.cc:2258] T f8ffc7146c364572aac283619f5acbed P b457496f7b59437eaec5d6efdd22b306 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:49.342127 17342 tablet_replica.cc:330] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280: stopping tablet replica
I20260811 11:50:49.342240 17342 raft_consensus.cc:2229] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:50:49.342571 17342 raft_consensus.cc:2258] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:49.343364 17342 ts_tablet_manager.cc:1905] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:49.343737 17347 ts_tablet_manager.cc:1905] T f8ffc7146c364572aac283619f5acbed P b457496f7b59437eaec5d6efdd22b306: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:49.346632 17342 ts_tablet_manager.cc:1918] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:50:49.346783 17342 log.cc:1196] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/f8ffc7146c364572aac283619f5acbed
I20260811 11:50:49.347131 17342 ts_tablet_manager.cc:1939] T f8ffc7146c364572aac283619f5acbed P fba11c35b7c549fd85bf67db6558f280: Deleting consensus metadata
I20260811 11:50:49.347998 16300 catalog_manager.cc:4841] TS fba11c35b7c549fd85bf67db6558f280 (127.3.221.67:37619): tablet f8ffc7146c364572aac283619f5acbed (table test-table [id=b219fa806411408c8888a565c66b6b91]) successfully deleted
I20260811 11:50:49.351130 17347 ts_tablet_manager.cc:1918] T f8ffc7146c364572aac283619f5acbed P b457496f7b59437eaec5d6efdd22b306: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:50:49.351212 17347 log.cc:1196] T f8ffc7146c364572aac283619f5acbed P b457496f7b59437eaec5d6efdd22b306: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-0/wal/wals/f8ffc7146c364572aac283619f5acbed
I20260811 11:50:49.351492 17347 ts_tablet_manager.cc:1939] T f8ffc7146c364572aac283619f5acbed P b457496f7b59437eaec5d6efdd22b306: Deleting consensus metadata
I20260811 11:50:49.352236 16297 catalog_manager.cc:4841] TS b457496f7b59437eaec5d6efdd22b306 (127.3.221.65:45263): tablet f8ffc7146c364572aac283619f5acbed (table test-table [id=b219fa806411408c8888a565c66b6b91]) successfully deleted
Aug 11 11:50:49 dist-test-slave-bndk krb5kdc[16255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 11:50:49 dist-test-slave-bndk krb5kdc[16255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:49 dist-test-slave-bndk krb5kdc[16255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
W20260811 11:50:49.371152 17357 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:49.363312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53260 (local address 127.3.221.126:46025)
0811 11:50:49.363603 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:49.363608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:49.363624 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:50:49.363740 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:49.363744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:49.363791 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:49.363867 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:49.364595 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.365144 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.366008 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.366256 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.366357 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:49.371005 (+ 4648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53260: BlockingRecv error: recv got EOF from 127.0.0.1:53260 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":102,"threads_started":1}
Aug 11 11:50:49 dist-test-slave-bndk krb5kdc[16255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM
I20260811 11:50:49.382670 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:49.371999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53264 (local address 127.3.221.126:46025)
0811 11:50:49.372122 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:49.372126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:49.372559 (+ 433us) server_negotiation.cc:407] Connection header received
0811 11:50:49.372679 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:49.372682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:49.372735 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:49.372813 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:49.373603 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.374437 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.375608 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.375971 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.376072 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:49.379006 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:49.379010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:49.379012 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:49.379043 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:49.380793 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:49.381306 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:49.381310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:49.381312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:49.381408 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:49.381676 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:49.381679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:49.381681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:49.381916 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:49.382050 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:49.382358 (+ 308us) server_negotiation.cc:299] Negotiation successful
0811 11:50:49.382498 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 11:50:49.385902 16318 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:50:49.386368 16318 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:50:49.391783 16584 tablet_service.cc:1467] Processing CreateTablet for tablet 59a8d2c4a77441c697315ffa6f369c4d (DEFAULT_TABLE table=test-table [id=37ec2abea1704b77a0deebe47d7eebc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:49.392068 17072 tablet_service.cc:1467] Processing CreateTablet for tablet 59a8d2c4a77441c697315ffa6f369c4d (DEFAULT_TABLE table=test-table [id=37ec2abea1704b77a0deebe47d7eebc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:49.392148 16584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a8d2c4a77441c697315ffa6f369c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:49.392328 17072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a8d2c4a77441c697315ffa6f369c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:49.394927 17367 tablet_bootstrap.cc:492] T 59a8d2c4a77441c697315ffa6f369c4d P b457496f7b59437eaec5d6efdd22b306: Bootstrap starting.
I20260811 11:50:49.395177 16841 tablet_service.cc:1467] Processing CreateTablet for tablet 59a8d2c4a77441c697315ffa6f369c4d (DEFAULT_TABLE table=test-table [id=37ec2abea1704b77a0deebe47d7eebc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:49.395499 16841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a8d2c4a77441c697315ffa6f369c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:49.396319 17367 tablet_bootstrap.cc:654] T 59a8d2c4a77441c697315ffa6f369c4d P b457496f7b59437eaec5d6efdd22b306: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:49.397367 17366 tablet_bootstrap.cc:492] T 59a8d2c4a77441c697315ffa6f369c4d P fba11c35b7c549fd85bf67db6558f280: Bootstrap starting.
I20260811 11:50:49.397962 17367 tablet_bootstrap.cc:492] T 59a8d2c4a77441c697315ffa6f369c4d P b457496f7b59437eaec5d6efdd22b306: No bootstrap required, opened a new log
I20260811 11:50:49.397976 17369 tablet_bootstrap.cc:492] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144: Bootstrap starting.
I20260811 11:50:49.398157 17367 ts_tablet_manager.cc:1397] T 59a8d2c4a77441c697315ffa6f369c4d P b457496f7b59437eaec5d6efdd22b306: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 11:50:49.398754 17367 raft_consensus.cc:348] T 59a8d2c4a77441c697315ffa6f369c4d P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:49.399000 17367 raft_consensus.cc:374] T 59a8d2c4a77441c697315ffa6f369c4d P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:49.399081 17367 raft_consensus.cc:729] T 59a8d2c4a77441c697315ffa6f369c4d P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b457496f7b59437eaec5d6efdd22b306, State: Initialized, Role: FOLLOWER
I20260811 11:50:49.399271 17369 tablet_bootstrap.cc:654] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:49.399860 17367 consensus_queue.cc:260] T 59a8d2c4a77441c697315ffa6f369c4d P b457496f7b59437eaec5d6efdd22b306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:49.400611 17367 ts_tablet_manager.cc:1428] T 59a8d2c4a77441c697315ffa6f369c4d P b457496f7b59437eaec5d6efdd22b306: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:50:49.401320 17369 tablet_bootstrap.cc:492] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144: No bootstrap required, opened a new log
I20260811 11:50:49.401423 17369 ts_tablet_manager.cc:1397] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 11:50:49.401722 17366 tablet_bootstrap.cc:654] T 59a8d2c4a77441c697315ffa6f369c4d P fba11c35b7c549fd85bf67db6558f280: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:49.401834 17369 raft_consensus.cc:348] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:49.401944 17369 raft_consensus.cc:374] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:49.401978 17369 raft_consensus.cc:729] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df141eeadc64b96ad0d38bd94281144, State: Initialized, Role: FOLLOWER
I20260811 11:50:49.402128 17369 consensus_queue.cc:260] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:49.402840 17369 ts_tablet_manager.cc:1428] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:49.406041 17366 tablet_bootstrap.cc:492] T 59a8d2c4a77441c697315ffa6f369c4d P fba11c35b7c549fd85bf67db6558f280: No bootstrap required, opened a new log
I20260811 11:50:49.406147 17366 ts_tablet_manager.cc:1397] T 59a8d2c4a77441c697315ffa6f369c4d P fba11c35b7c549fd85bf67db6558f280: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260811 11:50:49.406623 17366 raft_consensus.cc:348] T 59a8d2c4a77441c697315ffa6f369c4d P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:49.406750 17366 raft_consensus.cc:374] T 59a8d2c4a77441c697315ffa6f369c4d P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:49.406790 17366 raft_consensus.cc:729] T 59a8d2c4a77441c697315ffa6f369c4d P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fba11c35b7c549fd85bf67db6558f280, State: Initialized, Role: FOLLOWER
I20260811 11:50:49.406916 17366 consensus_queue.cc:260] T 59a8d2c4a77441c697315ffa6f369c4d P fba11c35b7c549fd85bf67db6558f280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:49.407282 17366 ts_tablet_manager.cc:1428] T 59a8d2c4a77441c697315ffa6f369c4d P fba11c35b7c549fd85bf67db6558f280: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:49.772434 17230 raft_consensus.cc:482] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:50:49.772591 17230 raft_consensus.cc:504] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:49.772995 17230 leader_election.cc:288] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b457496f7b59437eaec5d6efdd22b306 (127.3.221.65:45263), fba11c35b7c549fd85bf67db6558f280 (127.3.221.67:37619)
I20260811 11:50:49.773561 16625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a8d2c4a77441c697315ffa6f369c4d" candidate_uuid: "4df141eeadc64b96ad0d38bd94281144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b457496f7b59437eaec5d6efdd22b306" is_pre_election: true
I20260811 11:50:49.773559 17100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a8d2c4a77441c697315ffa6f369c4d" candidate_uuid: "4df141eeadc64b96ad0d38bd94281144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fba11c35b7c549fd85bf67db6558f280" is_pre_election: true
I20260811 11:50:49.773715 16625 raft_consensus.cc:2454] T 59a8d2c4a77441c697315ffa6f369c4d P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df141eeadc64b96ad0d38bd94281144 in term 0.
I20260811 11:50:49.773722 17100 raft_consensus.cc:2454] T 59a8d2c4a77441c697315ffa6f369c4d P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df141eeadc64b96ad0d38bd94281144 in term 0.
I20260811 11:50:49.774075 16774 leader_election.cc:302] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4df141eeadc64b96ad0d38bd94281144, b457496f7b59437eaec5d6efdd22b306, fba11c35b7c549fd85bf67db6558f280; no voters:
I20260811 11:50:49.774256 17230 raft_consensus.cc:2781] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:50:49.774320 17230 raft_consensus.cc:482] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:50:49.774365 17230 raft_consensus.cc:3037] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:49.775483 17230 raft_consensus.cc:504] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:49.775835 17230 leader_election.cc:288] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144 [CANDIDATE]: Term 1 election: Requested vote from peers b457496f7b59437eaec5d6efdd22b306 (127.3.221.65:45263), fba11c35b7c549fd85bf67db6558f280 (127.3.221.67:37619)
I20260811 11:50:49.776291 16625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a8d2c4a77441c697315ffa6f369c4d" candidate_uuid: "4df141eeadc64b96ad0d38bd94281144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b457496f7b59437eaec5d6efdd22b306"
I20260811 11:50:49.776427 16625 raft_consensus.cc:3037] T 59a8d2c4a77441c697315ffa6f369c4d P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:49.777415 16625 raft_consensus.cc:2454] T 59a8d2c4a77441c697315ffa6f369c4d P b457496f7b59437eaec5d6efdd22b306 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df141eeadc64b96ad0d38bd94281144 in term 1.
I20260811 11:50:49.777697 17100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a8d2c4a77441c697315ffa6f369c4d" candidate_uuid: "4df141eeadc64b96ad0d38bd94281144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fba11c35b7c549fd85bf67db6558f280"
I20260811 11:50:49.777734 16768 leader_election.cc:302] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df141eeadc64b96ad0d38bd94281144, b457496f7b59437eaec5d6efdd22b306; no voters:
I20260811 11:50:49.777822 17100 raft_consensus.cc:3037] T 59a8d2c4a77441c697315ffa6f369c4d P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:49.777916 17230 raft_consensus.cc:2781] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:49.778079 17230 raft_consensus.cc:686] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144 [term 1 LEADER]: Becoming Leader. State: Replica: 4df141eeadc64b96ad0d38bd94281144, State: Running, Role: LEADER
I20260811 11:50:49.778223 17230 consensus_queue.cc:237] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:49.779055 17100 raft_consensus.cc:2454] T 59a8d2c4a77441c697315ffa6f369c4d P fba11c35b7c549fd85bf67db6558f280 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df141eeadc64b96ad0d38bd94281144 in term 1.
I20260811 11:50:49.779946 16313 catalog_manager.cc:5495] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4df141eeadc64b96ad0d38bd94281144 (127.3.221.66). New cstate: current_term: 1 leader_uuid: "4df141eeadc64b96ad0d38bd94281144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } health_report { overall_health: UNKNOWN } } }
I20260811 11:50:49.853583 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:49.850325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41882 (local address 127.3.221.66:44233)
0811 11:50:49.850587 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:49.850592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:49.850606 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:50:49.850655 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:49.850657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:49.850705 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:49.850774 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:49.851039 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.851579 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.852442 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:49.852673 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:49.852761 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:49.853023 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:49.853102 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:49.853161 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:49.853285 (+ 124us) server_negotiation.cc:299] Negotiation successful
0811 11:50:49.853417 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":82,"threads_started":1}
I20260811 11:50:49.855756 16625 raft_consensus.cc:1261] T 59a8d2c4a77441c697315ffa6f369c4d P b457496f7b59437eaec5d6efdd22b306 [term 1 FOLLOWER]: Refusing update from remote peer 4df141eeadc64b96ad0d38bd94281144: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:49.855756 17100 raft_consensus.cc:1261] T 59a8d2c4a77441c697315ffa6f369c4d P fba11c35b7c549fd85bf67db6558f280 [term 1 FOLLOWER]: Refusing update from remote peer 4df141eeadc64b96ad0d38bd94281144: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:49.856248 17377 consensus_queue.cc:1035] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:49.856434 17230 consensus_queue.cc:1035] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:49.864706 16320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:49.864856 16320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:49.866832 16320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9: Sending DeleteTablet for 3 replicas of tablet 59a8d2c4a77441c697315ffa6f369c4d
I20260811 11:50:49.867498 17072 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a8d2c4a77441c697315ffa6f369c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:49 UTC) from {username='oryx', principal='oryx/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:56334
I20260811 11:50:49.867517 16584 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a8d2c4a77441c697315ffa6f369c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:49 UTC) from {username='oryx', principal='oryx/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:59490
I20260811 11:50:49.867517 16841 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a8d2c4a77441c697315ffa6f369c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:49 UTC) from {username='oryx', principal='oryx/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:41868
I20260811 11:50:49.867841 17401 tablet_replica.cc:330] T 59a8d2c4a77441c697315ffa6f369c4d P b457496f7b59437eaec5d6efdd22b306: stopping tablet replica
I20260811 11:50:49.867969 17401 raft_consensus.cc:2229] T 59a8d2c4a77441c697315ffa6f369c4d P b457496f7b59437eaec5d6efdd22b306 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:49.868106 17401 raft_consensus.cc:2258] T 59a8d2c4a77441c697315ffa6f369c4d P b457496f7b59437eaec5d6efdd22b306 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:49.868187 17405 tablet_replica.cc:330] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144: stopping tablet replica
I20260811 11:50:49.868317 17405 raft_consensus.cc:2229] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:50:49.868360 17404 tablet_replica.cc:330] T 59a8d2c4a77441c697315ffa6f369c4d P fba11c35b7c549fd85bf67db6558f280: stopping tablet replica
I20260811 11:50:49.868472 17404 raft_consensus.cc:2229] T 59a8d2c4a77441c697315ffa6f369c4d P fba11c35b7c549fd85bf67db6558f280 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:49.868530 17405 raft_consensus.cc:2258] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:49.868458 16320 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260811 11:50:49.868602 17404 raft_consensus.cc:2258] T 59a8d2c4a77441c697315ffa6f369c4d P fba11c35b7c549fd85bf67db6558f280 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260811 11:50:49.868857 16320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:50:49.869177 17405 ts_tablet_manager.cc:1905] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:49.870064 17401 ts_tablet_manager.cc:1905] T 59a8d2c4a77441c697315ffa6f369c4d P b457496f7b59437eaec5d6efdd22b306: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:49.870594 17404 ts_tablet_manager.cc:1905] T 59a8d2c4a77441c697315ffa6f369c4d P fba11c35b7c549fd85bf67db6558f280: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:49.871891 17405 ts_tablet_manager.cc:1918] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:50:49.871979 17405 log.cc:1196] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/59a8d2c4a77441c697315ffa6f369c4d
I20260811 11:50:49.872854 17401 ts_tablet_manager.cc:1918] T 59a8d2c4a77441c697315ffa6f369c4d P b457496f7b59437eaec5d6efdd22b306: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:50:49.872946 17401 log.cc:1196] T 59a8d2c4a77441c697315ffa6f369c4d P b457496f7b59437eaec5d6efdd22b306: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-0/wal/wals/59a8d2c4a77441c697315ffa6f369c4d
I20260811 11:50:49.873104 17405 ts_tablet_manager.cc:1939] T 59a8d2c4a77441c697315ffa6f369c4d P 4df141eeadc64b96ad0d38bd94281144: Deleting consensus metadata
I20260811 11:50:49.873289 17401 ts_tablet_manager.cc:1939] T 59a8d2c4a77441c697315ffa6f369c4d P b457496f7b59437eaec5d6efdd22b306: Deleting consensus metadata
I20260811 11:50:49.873800 17404 ts_tablet_manager.cc:1918] T 59a8d2c4a77441c697315ffa6f369c4d P fba11c35b7c549fd85bf67db6558f280: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:50:49.873991 16300 catalog_manager.cc:4841] TS 4df141eeadc64b96ad0d38bd94281144 (127.3.221.66:44233): tablet 59a8d2c4a77441c697315ffa6f369c4d (table test-table [id=37ec2abea1704b77a0deebe47d7eebc1]) successfully deleted
I20260811 11:50:49.874040 17404 log.cc:1196] T 59a8d2c4a77441c697315ffa6f369c4d P fba11c35b7c549fd85bf67db6558f280: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/59a8d2c4a77441c697315ffa6f369c4d
I20260811 11:50:49.874521 17404 ts_tablet_manager.cc:1939] T 59a8d2c4a77441c697315ffa6f369c4d P fba11c35b7c549fd85bf67db6558f280: Deleting consensus metadata
I20260811 11:50:49.874570 16841 tablet_service.cc:1467] Processing CreateTablet for tablet 555a543b30a44c17853235d9eba7ecd7 (DEFAULT_TABLE table=test-table [id=59dc647024f74f96961fbb9da729c751]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:49.874632 17072 tablet_service.cc:1467] Processing CreateTablet for tablet 555a543b30a44c17853235d9eba7ecd7 (DEFAULT_TABLE table=test-table [id=59dc647024f74f96961fbb9da729c751]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:49.874694 16584 tablet_service.cc:1467] Processing CreateTablet for tablet 555a543b30a44c17853235d9eba7ecd7 (DEFAULT_TABLE table=test-table [id=59dc647024f74f96961fbb9da729c751]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:50:49.874836 16841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 555a543b30a44c17853235d9eba7ecd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:49.874893 16584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 555a543b30a44c17853235d9eba7ecd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:49.874893 17072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 555a543b30a44c17853235d9eba7ecd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:49.875159 16300 catalog_manager.cc:4841] TS fba11c35b7c549fd85bf67db6558f280 (127.3.221.67:37619): tablet 59a8d2c4a77441c697315ffa6f369c4d (table test-table [id=37ec2abea1704b77a0deebe47d7eebc1]) successfully deleted
I20260811 11:50:49.877018 17367 tablet_bootstrap.cc:492] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306: Bootstrap starting.
I20260811 11:50:49.877094 17366 tablet_bootstrap.cc:492] T 555a543b30a44c17853235d9eba7ecd7 P fba11c35b7c549fd85bf67db6558f280: Bootstrap starting.
I20260811 11:50:49.877125 17369 tablet_bootstrap.cc:492] T 555a543b30a44c17853235d9eba7ecd7 P 4df141eeadc64b96ad0d38bd94281144: Bootstrap starting.
I20260811 11:50:49.877319 16297 catalog_manager.cc:4841] TS b457496f7b59437eaec5d6efdd22b306 (127.3.221.65:45263): tablet 59a8d2c4a77441c697315ffa6f369c4d (table test-table [id=37ec2abea1704b77a0deebe47d7eebc1]) successfully deleted
I20260811 11:50:49.877902 17366 tablet_bootstrap.cc:654] T 555a543b30a44c17853235d9eba7ecd7 P fba11c35b7c549fd85bf67db6558f280: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:49.877907 17369 tablet_bootstrap.cc:654] T 555a543b30a44c17853235d9eba7ecd7 P 4df141eeadc64b96ad0d38bd94281144: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:49.877929 17367 tablet_bootstrap.cc:654] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:49.878970 17366 tablet_bootstrap.cc:492] T 555a543b30a44c17853235d9eba7ecd7 P fba11c35b7c549fd85bf67db6558f280: No bootstrap required, opened a new log
I20260811 11:50:49.878970 17367 tablet_bootstrap.cc:492] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306: No bootstrap required, opened a new log
I20260811 11:50:49.879063 17366 ts_tablet_manager.cc:1397] T 555a543b30a44c17853235d9eba7ecd7 P fba11c35b7c549fd85bf67db6558f280: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:50:49.879063 17367 ts_tablet_manager.cc:1397] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:50:49.879192 17369 tablet_bootstrap.cc:492] T 555a543b30a44c17853235d9eba7ecd7 P 4df141eeadc64b96ad0d38bd94281144: No bootstrap required, opened a new log
I20260811 11:50:49.879276 17369 ts_tablet_manager.cc:1397] T 555a543b30a44c17853235d9eba7ecd7 P 4df141eeadc64b96ad0d38bd94281144: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 11:50:49.879465 17367 raft_consensus.cc:348] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:49.879465 17366 raft_consensus.cc:348] T 555a543b30a44c17853235d9eba7ecd7 P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:49.879590 17366 raft_consensus.cc:374] T 555a543b30a44c17853235d9eba7ecd7 P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:49.879590 17367 raft_consensus.cc:374] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:49.879623 17367 raft_consensus.cc:729] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b457496f7b59437eaec5d6efdd22b306, State: Initialized, Role: FOLLOWER
I20260811 11:50:49.879623 17366 raft_consensus.cc:729] T 555a543b30a44c17853235d9eba7ecd7 P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fba11c35b7c549fd85bf67db6558f280, State: Initialized, Role: FOLLOWER
I20260811 11:50:49.879617 17369 raft_consensus.cc:348] T 555a543b30a44c17853235d9eba7ecd7 P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:49.879735 17369 raft_consensus.cc:374] T 555a543b30a44c17853235d9eba7ecd7 P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:49.879774 17369 raft_consensus.cc:729] T 555a543b30a44c17853235d9eba7ecd7 P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df141eeadc64b96ad0d38bd94281144, State: Initialized, Role: FOLLOWER
I20260811 11:50:49.879758 17366 consensus_queue.cc:260] T 555a543b30a44c17853235d9eba7ecd7 P fba11c35b7c549fd85bf67db6558f280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:49.879756 17367 consensus_queue.cc:260] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:49.879906 17369 consensus_queue.cc:260] T 555a543b30a44c17853235d9eba7ecd7 P 4df141eeadc64b96ad0d38bd94281144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:49.880028 17366 ts_tablet_manager.cc:1428] T 555a543b30a44c17853235d9eba7ecd7 P fba11c35b7c549fd85bf67db6558f280: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:49.880028 17367 ts_tablet_manager.cc:1428] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:49.880160 17369 ts_tablet_manager.cc:1428] T 555a543b30a44c17853235d9eba7ecd7 P 4df141eeadc64b96ad0d38bd94281144: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 11:50:50.028116 17297 raft_consensus.cc:482] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:50:50.028283 17297 raft_consensus.cc:504] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:50.028741 17297 leader_election.cc:288] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4df141eeadc64b96ad0d38bd94281144 (127.3.221.66:44233), fba11c35b7c549fd85bf67db6558f280 (127.3.221.67:37619)
I20260811 11:50:50.029366 16875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "555a543b30a44c17853235d9eba7ecd7" candidate_uuid: "b457496f7b59437eaec5d6efdd22b306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df141eeadc64b96ad0d38bd94281144" is_pre_election: true
I20260811 11:50:50.029407 17100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "555a543b30a44c17853235d9eba7ecd7" candidate_uuid: "b457496f7b59437eaec5d6efdd22b306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fba11c35b7c549fd85bf67db6558f280" is_pre_election: true
I20260811 11:50:50.029524 16875 raft_consensus.cc:2454] T 555a543b30a44c17853235d9eba7ecd7 P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b457496f7b59437eaec5d6efdd22b306 in term 0.
I20260811 11:50:50.029596 17100 raft_consensus.cc:2454] T 555a543b30a44c17853235d9eba7ecd7 P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b457496f7b59437eaec5d6efdd22b306 in term 0.
I20260811 11:50:50.029884 16480 leader_election.cc:302] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df141eeadc64b96ad0d38bd94281144, b457496f7b59437eaec5d6efdd22b306; no voters:
I20260811 11:50:50.030059 17297 raft_consensus.cc:2781] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:50:50.030122 17297 raft_consensus.cc:482] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:50:50.030159 17297 raft_consensus.cc:3037] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:50.031188 17297 raft_consensus.cc:504] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:50.031517 17297 leader_election.cc:288] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306 [CANDIDATE]: Term 1 election: Requested vote from peers 4df141eeadc64b96ad0d38bd94281144 (127.3.221.66:44233), fba11c35b7c549fd85bf67db6558f280 (127.3.221.67:37619)
I20260811 11:50:50.031913 16875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "555a543b30a44c17853235d9eba7ecd7" candidate_uuid: "b457496f7b59437eaec5d6efdd22b306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df141eeadc64b96ad0d38bd94281144"
I20260811 11:50:50.031922 17100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "555a543b30a44c17853235d9eba7ecd7" candidate_uuid: "b457496f7b59437eaec5d6efdd22b306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fba11c35b7c549fd85bf67db6558f280"
I20260811 11:50:50.032028 16875 raft_consensus.cc:3037] T 555a543b30a44c17853235d9eba7ecd7 P 4df141eeadc64b96ad0d38bd94281144 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:50.032033 17100 raft_consensus.cc:3037] T 555a543b30a44c17853235d9eba7ecd7 P fba11c35b7c549fd85bf67db6558f280 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:50.032928 17100 raft_consensus.cc:2454] T 555a543b30a44c17853235d9eba7ecd7 P fba11c35b7c549fd85bf67db6558f280 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b457496f7b59437eaec5d6efdd22b306 in term 1.
I20260811 11:50:50.032975 16875 raft_consensus.cc:2454] T 555a543b30a44c17853235d9eba7ecd7 P 4df141eeadc64b96ad0d38bd94281144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b457496f7b59437eaec5d6efdd22b306 in term 1.
I20260811 11:50:50.033234 16480 leader_election.cc:302] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b457496f7b59437eaec5d6efdd22b306, fba11c35b7c549fd85bf67db6558f280; no voters:
I20260811 11:50:50.033413 17297 raft_consensus.cc:2781] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:50.033601 17297 raft_consensus.cc:686] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306 [term 1 LEADER]: Becoming Leader. State: Replica: b457496f7b59437eaec5d6efdd22b306, State: Running, Role: LEADER
I20260811 11:50:50.033742 17297 consensus_queue.cc:237] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } }
I20260811 11:50:50.035473 16320 catalog_manager.cc:5495] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306 reported cstate change: term changed from 0 to 1, leader changed from <none> to b457496f7b59437eaec5d6efdd22b306 (127.3.221.65). New cstate: current_term: 1 leader_uuid: "b457496f7b59437eaec5d6efdd22b306" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b457496f7b59437eaec5d6efdd22b306" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 } health_report { overall_health: UNKNOWN } } }
I20260811 11:50:50.081099 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:50.077939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59500 (local address 127.3.221.65:45263)
0811 11:50:50.078234 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:50.078240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:50.078253 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:50:50.078299 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:50.078302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:50.078349 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:50.078431 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:50.078699 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:50.079241 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:50.080108 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:50.080371 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:50.080462 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:50.080688 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:50.080771 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:50.080835 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:50.080917 (+ 82us) server_negotiation.cc:299] Negotiation successful
0811 11:50:50.080967 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":140,"threads_started":1}
I20260811 11:50:50.090741 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:50.087534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:45263 (local address 127.3.221.65:54177)
0811 11:50:50.087801 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:50.087814 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:50.087908 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:50.088090 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:50.088091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:50.088098 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 11:50:50.088183 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:50.088187 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:50.088893 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:50.088895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:50.089626 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:50.089636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:50.090088 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:50.090091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:50.090164 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:50.090192 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 11:50:50.090455 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 11:50:50.090459 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:50:50.090500 (+ 41us) client_negotiation.cc:240] Negotiation successful
0811 11:50:50.090570 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":96,"threads_started":1}
I20260811 11:50:50.090754 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:50.087626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:54177 (local address 127.3.221.65:45263)
0811 11:50:50.087783 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:50.087787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:50.087864 (+ 77us) server_negotiation.cc:407] Connection header received
0811 11:50:50.087971 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:50.087973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:50.088016 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:50.088078 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:50:50.088272 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:50.088772 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:50.089747 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:50.089961 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:50.090044 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:50.090322 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:50:50.090395 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:50:50.090446 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:50.090513 (+ 67us) server_negotiation.cc:299] Negotiation successful
0811 11:50:50.090561 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":60}
I20260811 11:50:50.096951 16875 raft_consensus.cc:1261] T 555a543b30a44c17853235d9eba7ecd7 P 4df141eeadc64b96ad0d38bd94281144 [term 1 FOLLOWER]: Refusing update from remote peer b457496f7b59437eaec5d6efdd22b306: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:50.096941 17100 raft_consensus.cc:1261] T 555a543b30a44c17853235d9eba7ecd7 P fba11c35b7c549fd85bf67db6558f280 [term 1 FOLLOWER]: Refusing update from remote peer b457496f7b59437eaec5d6efdd22b306: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:50:50.097347 17413 consensus_queue.cc:1035] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba11c35b7c549fd85bf67db6558f280" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:50.097577 17423 consensus_queue.cc:1035] T 555a543b30a44c17853235d9eba7ecd7 P b457496f7b59437eaec5d6efdd22b306 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4df141eeadc64b96ad0d38bd94281144" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:50:50.131471 16320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:50.131637 16320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:50:50.133787 16320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c4781c123434fb1912feba4dd41bae9: Sending DeleteTablet for 3 replicas of tablet 555a543b30a44c17853235d9eba7ecd7
I20260811 11:50:50.134485 16841 tablet_service.cc:1514] Processing DeleteTablet for tablet 555a543b30a44c17853235d9eba7ecd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:50 UTC) from {username='oryx', principal='oryx/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:41868
I20260811 11:50:50.134650 17072 tablet_service.cc:1514] Processing DeleteTablet for tablet 555a543b30a44c17853235d9eba7ecd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:50 UTC) from {username='oryx', principal='oryx/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:56334
I20260811 11:50:50.134840 17405 tablet_replica.cc:330] T 555a543b30a44c17853235d9eba7ecd7 P 4df141eeadc64b96ad0d38bd94281144: stopping tablet replica
I20260811 11:50:50.134975 17405 raft_consensus.cc:2229] T 555a543b30a44c17853235d9eba7ecd7 P 4df141eeadc64b96ad0d38bd94281144 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:50.135070 17404 tablet_replica.cc:330] T 555a543b30a44c17853235d9eba7ecd7 P fba11c35b7c549fd85bf67db6558f280: stopping tablet replica
I20260811 11:50:50.135098 17405 raft_consensus.cc:2258] T 555a543b30a44c17853235d9eba7ecd7 P 4df141eeadc64b96ad0d38bd94281144 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:50.135197 17404 raft_consensus.cc:2229] T 555a543b30a44c17853235d9eba7ecd7 P fba11c35b7c549fd85bf67db6558f280 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:50:50.135324 17404 raft_consensus.cc:2258] T 555a543b30a44c17853235d9eba7ecd7 P fba11c35b7c549fd85bf67db6558f280 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:50:50.135345 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 16450
I20260811 11:50:50.135494 16584 tablet_service.cc:1514] Processing DeleteTablet for tablet 555a543b30a44c17853235d9eba7ecd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:50:50 UTC) from {username='oryx', principal='oryx/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:59490
I20260811 11:50:50.146767 17405 ts_tablet_manager.cc:1905] T 555a543b30a44c17853235d9eba7ecd7 P 4df141eeadc64b96ad0d38bd94281144: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:50.149390 17404 ts_tablet_manager.cc:1905] T 555a543b30a44c17853235d9eba7ecd7 P fba11c35b7c549fd85bf67db6558f280: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:50:50.150218 17405 ts_tablet_manager.cc:1918] T 555a543b30a44c17853235d9eba7ecd7 P 4df141eeadc64b96ad0d38bd94281144: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:50:50.150292 17405 log.cc:1196] T 555a543b30a44c17853235d9eba7ecd7 P 4df141eeadc64b96ad0d38bd94281144: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/555a543b30a44c17853235d9eba7ecd7
I20260811 11:50:50.150516 17405 ts_tablet_manager.cc:1939] T 555a543b30a44c17853235d9eba7ecd7 P 4df141eeadc64b96ad0d38bd94281144: Deleting consensus metadata
I20260811 11:50:50.151958 17404 ts_tablet_manager.cc:1918] T 555a543b30a44c17853235d9eba7ecd7 P fba11c35b7c549fd85bf67db6558f280: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 11:50:50.152047 17404 log.cc:1196] T 555a543b30a44c17853235d9eba7ecd7 P fba11c35b7c549fd85bf67db6558f280: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/555a543b30a44c17853235d9eba7ecd7
I20260811 11:50:50.152287 17404 ts_tablet_manager.cc:1939] T 555a543b30a44c17853235d9eba7ecd7 P fba11c35b7c549fd85bf67db6558f280: Deleting consensus metadata
I20260811 11:50:50.153190 16300 catalog_manager.cc:4841] TS fba11c35b7c549fd85bf67db6558f280 (127.3.221.67:37619): tablet 555a543b30a44c17853235d9eba7ecd7 (table test-table [id=59dc647024f74f96961fbb9da729c751]) successfully deleted
I20260811 11:50:50.153674 16300 catalog_manager.cc:4841] TS 4df141eeadc64b96ad0d38bd94281144 (127.3.221.66:44233): tablet 555a543b30a44c17853235d9eba7ecd7 (table test-table [id=59dc647024f74f96961fbb9da729c751]) successfully deleted
W20260811 11:50:50.158285 16297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.221.65:45263 (error 108)
I20260811 11:50:50.158826 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 16743
I20260811 11:50:50.171864 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:50.159105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:45263 (local address 127.0.0.1:59502)
0811 11:50:50.171662 (+ 12557us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:50.171752 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.65:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12433,"thread_start_us":65,"threads_started":1}
W20260811 11:50:50.172870 16297 catalog_manager.cc:4584] TS b457496f7b59437eaec5d6efdd22b306 (127.3.221.65:45263): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 555a543b30a44c17853235d9eba7ecd7: Network error: Client connection negotiation failed: client connection to 127.3.221.65:45263: connect: Connection refused (error 111)
I20260811 11:50:50.179760 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 16963
I20260811 11:50:50.190991 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 16273
2026-08-11T11:50:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4058 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:50:50 dist-test-slave-bndk krb5kdc[17440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:50:50 dist-test-slave-bndk krb5kdc[17440](info): set up 2 sockets
Aug 11 11:50:50 dist-test-slave-bndk krb5kdc[17440](info): commencing operation
krb5kdc: starting...
W20260811 11:50:51.362622 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:50:51 dist-test-slave-bndk krb5kdc[17440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:50:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:50:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:50:51.496379 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:39265
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:35037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:39265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:51.616079 17462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:51.616380 17462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:51.616421 17462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:51.620484 17462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:50:51.620560 17462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:51.620579 17462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:51.620595 17462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:50:51.620609 17462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:50:51.626056 17462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:35037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:39265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:39265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:51.627131 17462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:51.628122 17462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:51.636865 17472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:51.636868 17481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:51.636858 17475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:51.636960 17462 server_base.cc:1034] running on GCE node
I20260811 11:50:51.637807 17462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:51.638989 17462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:51.640161 17462 hybrid_clock.cc:648] HybridClock initialized: now 1786449051640148 us; error 32 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-11T11:50:51Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1453 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:50:51 dist-test-slave-bndk krb5kdc[17490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:50:51 dist-test-slave-bndk krb5kdc[17490](info): set up 2 sockets
Aug 11 11:50:51 dist-test-slave-bndk krb5kdc[17490](info): commencing operation
krb5kdc: starting...
W20260811 11:50:52.814785 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:50:52 dist-test-slave-bndk krb5kdc[17490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:50:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:50:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:50:52.949110 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:40481
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:40481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:53.063973 17509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:53.064286 17509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:53.064350 17509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:53.068442 17509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:50:53.068522 17509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:53.068549 17509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:53.068570 17509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:50:53.068590 17509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:50:53.074215 17509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:40481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:40481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:53.075433 17509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:53.076550 17509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:53.085537 17522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:53.085765 17509 server_base.cc:1034] running on GCE node
W20260811 11:50:53.085541 17515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:53.085551 17516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:53.086517 17509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:53.087675 17509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:53.088884 17509 hybrid_clock.cc:648] HybridClock initialized: now 1786449053088866 us; error 28 us; skew 500 ppm
Aug 11 11:50:53 dist-test-slave-bndk krb5kdc[17490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449053, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:53.094760 17509 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:50:53.095825 17509 webserver.cc:460] Webserver started at http://127.3.221.126:37789/ using document root <none> and password file <none>
I20260811 11:50:53.096423 17509 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:53.096493 17509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:53.096707 17509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:53.098902 17509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "e319d621627a4fa69c7b38e6a0f29dba"
format_stamp: "Formatted at 2026-08-11 11:50:53 on dist-test-slave-bndk"
server_key: "e79c548914817ca3158b439ec65c36f6"
server_key_iv: "0a053d8ee84151773e67eba672f4a64b"
server_key_version: "encryptionkey@0"
I20260811 11:50:53.099478 17509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "e319d621627a4fa69c7b38e6a0f29dba"
format_stamp: "Formatted at 2026-08-11 11:50:53 on dist-test-slave-bndk"
server_key: "e79c548914817ca3158b439ec65c36f6"
server_key_iv: "0a053d8ee84151773e67eba672f4a64b"
server_key_version: "encryptionkey@0"
I20260811 11:50:53.103374 17509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 11:50:53.106007 17536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:53.107309 17509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 11:50:53.107436 17509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "e319d621627a4fa69c7b38e6a0f29dba"
format_stamp: "Formatted at 2026-08-11 11:50:53 on dist-test-slave-bndk"
server_key: "e79c548914817ca3158b439ec65c36f6"
server_key_iv: "0a053d8ee84151773e67eba672f4a64b"
server_key_version: "encryptionkey@0"
I20260811 11:50:53.107540 17509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:53.131145 17509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:53.134181 17509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:53.134382 17509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:53.141825 17509 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:40481
I20260811 11:50:53.141840 17637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:40481 every 8 connection(s)
I20260811 11:50:53.142861 17509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:50:53.145851 17639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:50:53.146973 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 17509
I20260811 11:50:53.147099 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:50:53.147379 3957 external_mini_cluster.cc:1428] Setting key cdb67ea33eab56893fa169b4ec761cdc
I20260811 11:50:53.152799 17639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba: Bootstrap starting.
I20260811 11:50:53.155541 17639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba: Neither blocks nor log segments found. Creating new log.
I20260811 11:50:53.156315 17639 log.cc:824] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba: Log is configured to *not* fsync() on all Append() calls
Aug 11 11:50:53 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:53.158658 17639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba: No bootstrap required, opened a new log
I20260811 11:50:53.162312 17645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:53.148852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34620 (local address 127.3.221.126:40481)
0811 11:50:53.149320 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:53.149340 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:53.149386 (+ 46us) server_negotiation.cc:407] Connection header received
0811 11:50:53.150217 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:53.150244 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:53.150689 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:53.151067 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:53.151810 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:53.153253 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:53.154142 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:53.154826 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:53.154980 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:53.157626 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:53.157632 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:53.157646 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:53.157674 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:53.159424 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:53.159902 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:53.159906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:53.159911 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:53.160020 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:53.160268 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:53.160270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:53.160271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:53.160669 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:53.160812 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:53.161192 (+ 380us) server_negotiation.cc:299] Negotiation successful
0811 11:50:53.161547 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":121,"threads_started":1}
I20260811 11:50:53.162695 17639 raft_consensus.cc:348] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e319d621627a4fa69c7b38e6a0f29dba" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40481 } }
I20260811 11:50:53.162925 17639 raft_consensus.cc:374] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:50:53.162981 17639 raft_consensus.cc:729] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e319d621627a4fa69c7b38e6a0f29dba, State: Initialized, Role: FOLLOWER
I20260811 11:50:53.163435 17639 consensus_queue.cc:260] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e319d621627a4fa69c7b38e6a0f29dba" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40481 } }
I20260811 11:50:53.163569 17639 raft_consensus.cc:388] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:50:53.163636 17639 raft_consensus.cc:482] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:50:53.163733 17639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:50:53.164721 17639 raft_consensus.cc:504] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e319d621627a4fa69c7b38e6a0f29dba" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40481 } }
I20260811 11:50:53.165055 17639 leader_election.cc:302] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e319d621627a4fa69c7b38e6a0f29dba; no voters:
I20260811 11:50:53.165380 17639 leader_election.cc:288] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:50:53.165550 17650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:50:53.165767 17650 raft_consensus.cc:686] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [term 1 LEADER]: Becoming Leader. State: Replica: e319d621627a4fa69c7b38e6a0f29dba, State: Running, Role: LEADER
I20260811 11:50:53.166095 17650 consensus_queue.cc:237] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e319d621627a4fa69c7b38e6a0f29dba" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40481 } }
I20260811 11:50:53.166610 17639 sys_catalog.cc:564] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:50:53.167692 17653 sys_catalog.cc:455] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [sys.catalog]: SysCatalogTable state changed. Reason: New leader e319d621627a4fa69c7b38e6a0f29dba. Latest consensus state: current_term: 1 leader_uuid: "e319d621627a4fa69c7b38e6a0f29dba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e319d621627a4fa69c7b38e6a0f29dba" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40481 } } }
I20260811 11:50:53.167825 17653 sys_catalog.cc:458] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:53.168304 17664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:50:53.168320 17652 sys_catalog.cc:455] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e319d621627a4fa69c7b38e6a0f29dba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e319d621627a4fa69c7b38e6a0f29dba" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40481 } } }
I20260811 11:50:53.168438 17652 sys_catalog.cc:458] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:53.171728 17664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:50:53.177903 17664 catalog_manager.cc:1347] Generated new cluster ID: 902f44a4f95f43d89ed5178ce045be3d
I20260811 11:50:53.177984 17664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:50:53.188669 17664 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:50:53.189953 17664 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:50:53.197554 17664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba: Generated new TSK 0
I20260811 11:50:53.198307 17664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
I20260811 11:50:53.230244 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:40481
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:53.369740 17686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:53.370024 17686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:53.370088 17686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:53.374228 17686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:53.374306 17686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:53.374388 17686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:50:53.380030 17686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:40481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:53.381188 17686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:53.382294 17686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:53.392300 17701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:53.392306 17694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:53.392364 17693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:53.392634 17686 server_base.cc:1034] running on GCE node
I20260811 11:50:53.393128 17686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:53.393836 17686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:53.395036 17686 hybrid_clock.cc:648] HybridClock initialized: now 1786449053394997 us; error 49 us; skew 500 ppm
Aug 11 11:50:53 dist-test-slave-bndk krb5kdc[17490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449053, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:53.402544 17686 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:50:53.403571 17686 webserver.cc:460] Webserver started at http://127.3.221.65:45637/ using document root <none> and password file <none>
I20260811 11:50:53.404161 17686 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:53.404227 17686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:53.404436 17686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:53.406656 17686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "2978a1325cf34fb8a9e60f1c4fa1d380"
format_stamp: "Formatted at 2026-08-11 11:50:53 on dist-test-slave-bndk"
server_key: "5c7581f5c00cf203f64fccff848d57b7"
server_key_iv: "9d331007478038633c939dcc3d06d799"
server_key_version: "encryptionkey@0"
I20260811 11:50:53.407219 17686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "2978a1325cf34fb8a9e60f1c4fa1d380"
format_stamp: "Formatted at 2026-08-11 11:50:53 on dist-test-slave-bndk"
server_key: "5c7581f5c00cf203f64fccff848d57b7"
server_key_iv: "9d331007478038633c939dcc3d06d799"
server_key_version: "encryptionkey@0"
I20260811 11:50:53.411290 17686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260811 11:50:53.413993 17710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:53.415416 17686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:50:53.415534 17686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "2978a1325cf34fb8a9e60f1c4fa1d380"
format_stamp: "Formatted at 2026-08-11 11:50:53 on dist-test-slave-bndk"
server_key: "5c7581f5c00cf203f64fccff848d57b7"
server_key_iv: "9d331007478038633c939dcc3d06d799"
server_key_version: "encryptionkey@0"
I20260811 11:50:53.415658 17686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:53.439641 17686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:53.442911 17686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:53.443099 17686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:53.443728 17686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:53.444617 17686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:53.444664 17686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:53.444708 17686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:53.444736 17686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:53.453502 17686 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:35549
I20260811 11:50:53.453531 17896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:35549 every 8 connection(s)
I20260811 11:50:53.454424 17686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:50:53.460907 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 17686
I20260811 11:50:53.461046 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:50:53.461380 3957 external_mini_cluster.cc:1428] Setting key 765fabdfea26d829dc65e6d5aea77d9d
Aug 11 11:50:53 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449053, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:53.469022 17645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:53.456292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:42099 (local address 127.3.221.126:40481)
0811 11:50:53.456466 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:53.456471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:53.457369 (+ 898us) server_negotiation.cc:407] Connection header received
0811 11:50:53.458611 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:53.458614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:53.458670 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:53.458745 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:53.460034 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:53.460589 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:53.462027 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:53.462307 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:53.462399 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:53.465035 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:53.465038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:53.465040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:53.465067 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:53.466500 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:53.467133 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:53.467137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:53.467138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:53.467245 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:53.467664 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:53.467667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:53.467668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:53.467875 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:53.467959 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:53.468724 (+ 765us) server_negotiation.cc:299] Negotiation successful
0811 11:50:53.468888 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 11:50:53.470237 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:53.456691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.65:42099)
0811 11:50:53.457161 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:53.457206 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:53.458403 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:53.458967 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:53.458974 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:53.459353 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:53.459867 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:53.459882 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:53.460753 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:53.460756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:53.461903 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:53.461915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:53.462478 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:53.462482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:53.462677 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:53.463361 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:53.463379 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:53.464854 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:53.466632 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:53.466638 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:53.466657 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:53.467035 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:53.467355 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:53.467358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:53.467360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:53.467561 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:53.467981 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:53.467988 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:53.468381 (+ 393us) client_negotiation.cc:769] Sending connection context
0811 11:50:53.468637 (+ 256us) client_negotiation.cc:240] Negotiation successful
0811 11:50:53.469035 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":132,"threads_started":1}
I20260811 11:50:53.471532 17898 heartbeater.cc:347] Connected to a master server at 127.3.221.126:40481
I20260811 11:50:53.471781 17898 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:53.472285 17898 heartbeater.cc:511] Master 127.3.221.126:40481 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.221.66@KRBTEST.COM; defaulting to no policy
I20260811 11:50:53.473862 17569 ts_manager.cc:194] Registered new tserver with Master: 2978a1325cf34fb8a9e60f1c4fa1d380 (127.3.221.65:35549)
I20260811 11:50:53.475116 17569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:42099
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
I20260811 11:50:53.492374 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:40481
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:53.607967 17904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:53.608246 17904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:53.608289 17904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:53.612452 17904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:53.612524 17904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:53.612583 17904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:50:53.617924 17904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:40481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:53.619086 17904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:53.620227 17904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:53.632097 17912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:53.632220 17914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:53.632097 17911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:53.632665 17904 server_base.cc:1034] running on GCE node
I20260811 11:50:53.633158 17904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:53.633948 17904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:53.635140 17904 hybrid_clock.cc:648] HybridClock initialized: now 1786449053635107 us; error 43 us; skew 500 ppm
Aug 11 11:50:53 dist-test-slave-bndk krb5kdc[17490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449053, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:53.638291 17904 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:50:53.639226 17904 webserver.cc:460] Webserver started at http://127.3.221.66:35049/ using document root <none> and password file <none>
I20260811 11:50:53.639775 17904 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:53.639837 17904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:53.640018 17904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:53.642197 17904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "ae6d2563be9049248f1571c322e1f407"
format_stamp: "Formatted at 2026-08-11 11:50:53 on dist-test-slave-bndk"
server_key: "a019bfe1bea8108155c13404d5c35aed"
server_key_iv: "1056c7977b1bb83e836463f9637ce154"
server_key_version: "encryptionkey@0"
I20260811 11:50:53.642728 17904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "ae6d2563be9049248f1571c322e1f407"
format_stamp: "Formatted at 2026-08-11 11:50:53 on dist-test-slave-bndk"
server_key: "a019bfe1bea8108155c13404d5c35aed"
server_key_iv: "1056c7977b1bb83e836463f9637ce154"
server_key_version: "encryptionkey@0"
I20260811 11:50:53.646581 17904 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:50:53.649037 17923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:53.650344 17904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 11:50:53.650470 17904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "ae6d2563be9049248f1571c322e1f407"
format_stamp: "Formatted at 2026-08-11 11:50:53 on dist-test-slave-bndk"
server_key: "a019bfe1bea8108155c13404d5c35aed"
server_key_iv: "1056c7977b1bb83e836463f9637ce154"
server_key_version: "encryptionkey@0"
I20260811 11:50:53.650581 17904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:53.661514 17904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:53.664696 17904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:53.664884 17904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:53.665609 17904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:53.666522 17904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:53.666571 17904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:53.666625 17904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:53.666651 17904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:53.675652 17904 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:36879
I20260811 11:50:53.675694 18119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:36879 every 8 connection(s)
I20260811 11:50:53.676568 17904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:50:53.680047 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 17904
I20260811 11:50:53.680189 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:50:53.680502 3957 external_mini_cluster.cc:1428] Setting key 8a3395cb94823aab7feb1e2effe970c7
Aug 11 11:50:53 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449053, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.221.67@KRBTEST.COM; defaulting to no policy
I20260811 11:50:53.691406 17645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:53.678459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:34659 (local address 127.3.221.126:40481)
0811 11:50:53.678581 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:53.678584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:53.679452 (+ 868us) server_negotiation.cc:407] Connection header received
0811 11:50:53.681029 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:53.681033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:53.681093 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:53.681166 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:53.682638 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:53.683147 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:53.684582 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:53.684895 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:53.685001 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:53.687533 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:53.687536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:53.687538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:53.687564 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:53.688869 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:53.689449 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:53.689452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:53.689453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:53.689533 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:53.689981 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:53.689985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:53.689987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:53.690267 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:53.690381 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:53.691067 (+ 686us) server_negotiation.cc:299] Negotiation successful
0811 11:50:53.691208 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 11:50:53.692505 18125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:53.678837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.66:34659)
0811 11:50:53.679292 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:53.679338 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:53.680768 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:53.681389 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:53.681397 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:53.681767 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:53.682483 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:53.682507 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:53.683272 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:53.683276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:53.684458 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:53.684470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:53.685078 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:53.685081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:53.685278 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:53.685881 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:53.685899 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:53.687359 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:53.688987 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:53.688993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:53.689011 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:53.689345 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:53.689619 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:53.689622 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:53.689624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:53.689812 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:53.690385 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:53.690393 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:53.690744 (+ 351us) client_negotiation.cc:769] Sending connection context
0811 11:50:53.690971 (+ 227us) client_negotiation.cc:240] Negotiation successful
0811 11:50:53.691372 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.67@KRBTEST.COM" created.
I20260811 11:50:53.694681 18122 heartbeater.cc:347] Connected to a master server at 127.3.221.126:40481
I20260811 11:50:53.695031 18122 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:53.695670 18122 heartbeater.cc:511] Master 127.3.221.126:40481 requested a full tablet report, sending...
I20260811 11:50:53.696956 17569 ts_manager.cc:194] Registered new tserver with Master: ae6d2563be9049248f1571c322e1f407 (127.3.221.66:36879)
I20260811 11:50:53.697785 17569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:34659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
I20260811 11:50:53.712428 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:40481
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:53.827701 18131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:53.828070 18131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:53.828141 18131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:53.832369 18131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:53.832436 18131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:53.832496 18131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:50:53.837879 18131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:40481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:53.838959 18131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:53.839993 18131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:53.849728 18140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:53.849718 18137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:53.849723 18138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:53.850045 18131 server_base.cc:1034] running on GCE node
I20260811 11:50:53.850459 18131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:53.851101 18131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:53.852277 18131 hybrid_clock.cc:648] HybridClock initialized: now 1786449053852246 us; error 43 us; skew 500 ppm
Aug 11 11:50:53 dist-test-slave-bndk krb5kdc[17490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449053, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:53.855170 18131 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:50:53.856108 18131 webserver.cc:460] Webserver started at http://127.3.221.67:46425/ using document root <none> and password file <none>
I20260811 11:50:53.856678 18131 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:53.856750 18131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:53.856963 18131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:50:53.859124 18131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "fe1e0590b2824091a71eb7aec531ddcd"
format_stamp: "Formatted at 2026-08-11 11:50:53 on dist-test-slave-bndk"
server_key: "6c8f3681df833072fbb49fd558ab902b"
server_key_iv: "03fe3cdfd76fe12a14e91fc06e7fcca2"
server_key_version: "encryptionkey@0"
I20260811 11:50:53.859690 18131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "fe1e0590b2824091a71eb7aec531ddcd"
format_stamp: "Formatted at 2026-08-11 11:50:53 on dist-test-slave-bndk"
server_key: "6c8f3681df833072fbb49fd558ab902b"
server_key_iv: "03fe3cdfd76fe12a14e91fc06e7fcca2"
server_key_version: "encryptionkey@0"
I20260811 11:50:53.863659 18131 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 11:50:53.866325 18150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:53.867751 18131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:50:53.867892 18131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "fe1e0590b2824091a71eb7aec531ddcd"
format_stamp: "Formatted at 2026-08-11 11:50:53 on dist-test-slave-bndk"
server_key: "6c8f3681df833072fbb49fd558ab902b"
server_key_iv: "03fe3cdfd76fe12a14e91fc06e7fcca2"
server_key_version: "encryptionkey@0"
I20260811 11:50:53.868001 18131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:53.884496 18131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:53.887813 18131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:53.887995 18131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:53.888603 18131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:53.889557 18131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:53.889608 18131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:53.889667 18131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:53.889691 18131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:53.898649 18131 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:33277
I20260811 11:50:53.898741 18338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:33277 every 8 connection(s)
I20260811 11:50:53.899618 18131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:50:53.900254 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 18131
I20260811 11:50:53.900399 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:50:53.900679 3957 external_mini_cluster.cc:1428] Setting key 46a51cabf5a91a58d19eb5ff7281ba01
Aug 11 11:50:53 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449053, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:50:53.912900 17645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:53.901508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:35469 (local address 127.3.221.126:40481)
0811 11:50:53.901630 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:53.901635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:53.902806 (+ 1171us) server_negotiation.cc:407] Connection header received
0811 11:50:53.904009 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:53.904012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:53.904058 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:53.904123 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:53.905278 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:53.905805 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:53.906913 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:53.907132 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:53.907211 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:53.909811 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:53.909815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:53.909817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:53.909843 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:53.910958 (+ 1115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:53.911471 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:53.911475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:53.911476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:53.911582 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:53.911926 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:53.911928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:53.911929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:53.912111 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:53.912188 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:53.912657 (+ 469us) server_negotiation.cc:299] Negotiation successful
0811 11:50:53.912773 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 11:50:53.914265 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:53.902009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.67:35469)
0811 11:50:53.902538 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:53.902594 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:53.903817 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:53.904264 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:53.904272 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:53.904629 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:53.905130 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:53.905144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:53.905907 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:53.905910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:53.906805 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:53.906815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:53.907297 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:53.907301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:53.907466 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:53.908004 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:53.908021 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:53.909632 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:50:53.911057 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:53.911062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:53.911075 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:53.911357 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:53.911657 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:50:53.911659 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:50:53.911660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:50:53.911833 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:50:53.912198 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:50:53.912203 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:50:53.912457 (+ 254us) client_negotiation.cc:769] Sending connection context
0811 11:50:53.912621 (+ 164us) client_negotiation.cc:240] Negotiation successful
0811 11:50:53.912888 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":150,"threads_started":1}
I20260811 11:50:53.915406 18348 heartbeater.cc:347] Connected to a master server at 127.3.221.126:40481
I20260811 11:50:53.915655 18348 heartbeater.cc:464] Registering TS with master...
I20260811 11:50:53.916182 18348 heartbeater.cc:511] Master 127.3.221.126:40481 requested a full tablet report, sending...
I20260811 11:50:53.917336 17569 ts_manager.cc:194] Registered new tserver with Master: fe1e0590b2824091a71eb7aec531ddcd (127.3.221.67:33277)
I20260811 11:50:53.917922 17569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:35469
I20260811 11:50:53.926649 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.65.keytab.
Entry for principal oryx/127.3.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.65.keytab.
WARNING: no policy specified for oryx/127.3.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.66.keytab.
Entry for principal oryx/127.3.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.66.keytab.
WARNING: no policy specified for oryx/127.3.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.67.keytab.
Entry for principal oryx/127.3.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.67.keytab.
I20260811 11:50:54.007978 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 17686
I20260811 11:50:54.042253 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 17904
I20260811 11:50:54.054416 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 18131
I20260811 11:50:54.063994 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 17509
I20260811 11:50:54.073773 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:40481
--webserver_interface=127.3.221.126
--webserver_port=37789
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:40481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:50:54.191908 18357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:54.192220 18357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:54.192263 18357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:54.196393 18357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:50:54.196476 18357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:54.196496 18357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:54.196511 18357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:50:54.196525 18357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:50:54.202301 18357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:40481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:40481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=37789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:54.203483 18357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:54.204573 18357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:54.213311 18367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:54.213320 18363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:54.213335 18357 server_base.cc:1034] running on GCE node
W20260811 11:50:54.213553 18365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:54.214187 18357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:54.215322 18357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:54.216526 18357 hybrid_clock.cc:648] HybridClock initialized: now 1786449054216493 us; error 47 us; skew 500 ppm
Aug 11 11:50:54 dist-test-slave-bndk krb5kdc[17490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449054, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:54.219651 18357 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:50:54.220693 18357 webserver.cc:460] Webserver started at http://127.3.221.126:37789/ using document root <none> and password file <none>
I20260811 11:50:54.221345 18357 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:54.221441 18357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:54.225888 18357 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 11:50:54.228245 18381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:54.229635 18357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:50:54.229764 18357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "e319d621627a4fa69c7b38e6a0f29dba"
format_stamp: "Formatted at 2026-08-11 11:50:53 on dist-test-slave-bndk"
server_key: "e79c548914817ca3158b439ec65c36f6"
server_key_iv: "0a053d8ee84151773e67eba672f4a64b"
server_key_version: "encryptionkey@0"
I20260811 11:50:54.230376 18357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:54.270754 18357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:54.274330 18357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:54.274514 18357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:54.282088 18357 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:40481
I20260811 11:50:54.282157 18481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:40481 every 8 connection(s)
I20260811 11:50:54.283242 18357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:50:54.287963 18483 sys_catalog.cc:263] Verifying existing consensus state
I20260811 11:50:54.289981 18483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba: Bootstrap starting.
I20260811 11:50:54.291679 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 18357
I20260811 11:50:54.292227 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:35549
--local_ip_for_outbound_sockets=127.3.221.65
--tserver_master_addrs=127.3.221.126:40481
--webserver_port=45637
--webserver_interface=127.3.221.65
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.221.65
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:50:54.312451 18483 log.cc:824] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba: Log is configured to *not* fsync() on all Append() calls
I20260811 11:50:54.322214 18483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:50:54.322712 18483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba: Bootstrap complete.
I20260811 11:50:54.328764 18483 raft_consensus.cc:348] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e319d621627a4fa69c7b38e6a0f29dba" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40481 } }
I20260811 11:50:54.329382 18483 raft_consensus.cc:729] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e319d621627a4fa69c7b38e6a0f29dba, State: Initialized, Role: FOLLOWER
I20260811 11:50:54.329995 18483 consensus_queue.cc:260] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e319d621627a4fa69c7b38e6a0f29dba" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40481 } }
I20260811 11:50:54.330156 18483 raft_consensus.cc:388] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:50:54.330240 18483 raft_consensus.cc:482] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:50:54.330341 18483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [term 1 FOLLOWER]: Advancing to term 2
I20260811 11:50:54.333128 18483 raft_consensus.cc:504] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e319d621627a4fa69c7b38e6a0f29dba" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40481 } }
I20260811 11:50:54.333567 18483 leader_election.cc:302] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e319d621627a4fa69c7b38e6a0f29dba; no voters:
I20260811 11:50:54.333956 18483 leader_election.cc:288] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 11:50:54.334101 18491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [term 2 FOLLOWER]: Leader election won for term 2
I20260811 11:50:54.334456 18491 raft_consensus.cc:686] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [term 2 LEADER]: Becoming Leader. State: Replica: e319d621627a4fa69c7b38e6a0f29dba, State: Running, Role: LEADER
I20260811 11:50:54.334793 18491 consensus_queue.cc:237] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e319d621627a4fa69c7b38e6a0f29dba" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40481 } }
I20260811 11:50:54.335413 18483 sys_catalog.cc:564] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:50:54.335798 18491 sys_catalog.cc:455] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [sys.catalog]: SysCatalogTable state changed. Reason: New leader e319d621627a4fa69c7b38e6a0f29dba. Latest consensus state: current_term: 2 leader_uuid: "e319d621627a4fa69c7b38e6a0f29dba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e319d621627a4fa69c7b38e6a0f29dba" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40481 } } }
I20260811 11:50:54.335923 18491 sys_catalog.cc:458] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:54.336328 18492 sys_catalog.cc:455] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e319d621627a4fa69c7b38e6a0f29dba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e319d621627a4fa69c7b38e6a0f29dba" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40481 } } }
I20260811 11:50:54.336334 18496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:50:54.337509 18492 sys_catalog.cc:458] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba [sys.catalog]: This master's current role is: LEADER
I20260811 11:50:54.339879 18496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:50:54.340873 18496 catalog_manager.cc:1259] Loaded cluster ID: 902f44a4f95f43d89ed5178ce045be3d
I20260811 11:50:54.340938 18496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:50:54.343407 18496 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:50:54.344161 18496 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e319d621627a4fa69c7b38e6a0f29dba: Loaded TSK: 0
I20260811 11:50:54.345465 18496 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 11:50:54.427106 18485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:54.427380 18485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:54.427443 18485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:54.431581 18485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:54.431648 18485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:54.431726 18485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:50:54.437443 18485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.65.keytab
--principal=oryx/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:35549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=45637
--tserver_master_addrs=127.3.221.126:40481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:54.438501 18485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:54.439534 18485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:54.450663 18526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:54.450732 18485 server_base.cc:1034] running on GCE node
W20260811 11:50:54.450947 18527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:54.450670 18530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:54.451409 18485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:54.452060 18485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:54.453248 18485 hybrid_clock.cc:648] HybridClock initialized: now 1786449054453195 us; error 52 us; skew 500 ppm
Aug 11 11:50:54 dist-test-slave-bndk krb5kdc[17490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449054, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:54.456058 18485 init.cc:377] Logged in from keytab as oryx/127.3.221.65@KRBTEST.COM (short username oryx)
I20260811 11:50:54.457013 18485 webserver.cc:460] Webserver started at http://127.3.221.65:45637/ using document root <none> and password file <none>
I20260811 11:50:54.457638 18485 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:54.457721 18485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:54.461983 18485 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:50:54.464362 18542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:54.465863 18485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:50:54.466008 18485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "2978a1325cf34fb8a9e60f1c4fa1d380"
format_stamp: "Formatted at 2026-08-11 11:50:53 on dist-test-slave-bndk"
server_key: "5c7581f5c00cf203f64fccff848d57b7"
server_key_iv: "9d331007478038633c939dcc3d06d799"
server_key_version: "encryptionkey@0"
I20260811 11:50:54.466662 18485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:54.482918 18485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:54.485956 18485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:54.486145 18485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:54.486763 18485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:54.487623 18485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:54.487671 18485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:54.487726 18485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:54.487754 18485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:54.496398 18485 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:35549
I20260811 11:50:54.496423 18736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:35549 every 8 connection(s)
I20260811 11:50:54.497459 18485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:50:54.501385 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 18485
I20260811 11:50:54.501820 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:36879
--local_ip_for_outbound_sockets=127.3.221.66
--tserver_master_addrs=127.3.221.126:40481
--webserver_port=35049
--webserver_interface=127.3.221.66
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.221.66
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 11:50:54 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:54 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:54.511264 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:54.499794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.65:52423)
0811 11:50:54.500268 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:54.500321 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:54.501615 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:54.502942 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:54.502949 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:54.503306 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:54.503925 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:54.503946 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.505361 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.505364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:54.506263 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:54.506274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.507330 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.507333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:54.507482 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:54.508076 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:54.508093 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:54.510460 (+ 2367us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":122,"threads_started":1}
W20260811 11:50:54.511750 18740 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:54.499686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:52423 (local address 127.3.221.126:40481)
0811 11:50:54.500143 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:54.500160 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:54.500427 (+ 267us) server_negotiation.cc:407] Connection header received
0811 11:50:54.502070 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:54.502086 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:54.502498 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:54.502873 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:54.504111 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.505209 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.506400 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.507164 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.507359 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:54.510777 (+ 3418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.65:52423: BlockingRecv error: recv got EOF from 127.3.221.65:52423 (error 108)
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":114,"threads_started":1}
W20260811 11:50:54.511876 18737 heartbeater.cc:644] Failed to heartbeat to 127.3.221.126:40481 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.221.126:40481: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Aug 11 11:50:54 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:54 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:54.517115 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:54.512412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.65:45495)
0811 11:50:54.512563 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:54.512577 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:54.512652 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:54.512877 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:54.512880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:54.513042 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:54.513133 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:54.513139 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.513872 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.513875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:54.514410 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:54.514419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.514826 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.514830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:54.514930 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:54.515388 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:54.515398 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:54.516992 (+ 1594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93}
W20260811 11:50:54.517257 18740 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:54.512524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:45495 (local address 127.3.221.126:40481)
0811 11:50:54.512645 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:54.512649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:54.512663 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:50:54.512737 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:54.512740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:54.512783 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:54.512850 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:54.513261 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.513769 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.514519 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.514729 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.514816 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:54.517137 (+ 2321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.65:45495: BlockingRecv error: recv got EOF from 127.3.221.65:45495 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 11 11:50:54 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:54 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:54.522212 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:54.517651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.65:41789)
0811 11:50:54.517848 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:54.517861 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:54.517947 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:54.518158 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:54.518160 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:54.518301 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:54.518367 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:54.518372 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.519038 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.519041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:54.519543 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:54.519550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.519952 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.519955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:54.520029 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:54.520458 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:54.520468 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:54.522085 (+ 1617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136}
W20260811 11:50:54.522293 18740 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:54.517741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:41789 (local address 127.3.221.126:40481)
0811 11:50:54.517835 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:54.517838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:54.517901 (+ 63us) server_negotiation.cc:407] Connection header received
0811 11:50:54.518026 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:54.518029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:54.518073 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:54.518134 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:54.518471 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.518942 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.519644 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.519852 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.519934 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:54.522202 (+ 2268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.65:41789: BlockingRecv error: recv got EOF from 127.3.221.65:41789 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260811 11:50:54.522461 18737 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 11:50:54.622361 18742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:54.622663 18742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:54.622709 18742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:54.626888 18742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:54.626973 18742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:54.627036 18742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:50:54.632563 18742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.66.keytab
--principal=oryx/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:36879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=35049
--tserver_master_addrs=127.3.221.126:40481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:54.633821 18742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:54.634944 18742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:50:54.645001 18742 server_base.cc:1034] running on GCE node
W20260811 11:50:54.644945 18763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:54.644953 18759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:54.644945 18758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:54.645862 18742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:54.646600 18742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:54.647776 18742 hybrid_clock.cc:648] HybridClock initialized: now 1786449054647755 us; error 35 us; skew 500 ppm
Aug 11 11:50:54 dist-test-slave-bndk krb5kdc[17490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449054, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:54.650664 18742 init.cc:377] Logged in from keytab as oryx/127.3.221.66@KRBTEST.COM (short username oryx)
I20260811 11:50:54.651849 18742 webserver.cc:460] Webserver started at http://127.3.221.66:35049/ using document root <none> and password file <none>
I20260811 11:50:54.652477 18742 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:54.652559 18742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:54.656613 18742 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260811 11:50:54.658816 18775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:54.660148 18742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 11:50:54.660269 18742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "ae6d2563be9049248f1571c322e1f407"
format_stamp: "Formatted at 2026-08-11 11:50:53 on dist-test-slave-bndk"
server_key: "a019bfe1bea8108155c13404d5c35aed"
server_key_iv: "1056c7977b1bb83e836463f9637ce154"
server_key_version: "encryptionkey@0"
I20260811 11:50:54.660815 18742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:54.671236 18742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:54.674556 18742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:54.674731 18742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:54.675318 18742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:54.676172 18742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:54.676219 18742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:54.676257 18742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:54.676271 18742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:54.685309 18742 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:36879
I20260811 11:50:54.685416 18958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:36879 every 8 connection(s)
I20260811 11:50:54.686431 18742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:50:54.689877 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 18742
I20260811 11:50:54.690486 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:33277
--local_ip_for_outbound_sockets=127.3.221.67
--tserver_master_addrs=127.3.221.126:40481
--webserver_port=46425
--webserver_interface=127.3.221.67
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.221.67
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 11:50:54 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:54 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
W20260811 11:50:54.698943 18740 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:54.688369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:48683 (local address 127.3.221.126:40481)
0811 11:50:54.688499 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:54.688503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:54.689455 (+ 952us) server_negotiation.cc:407] Connection header received
0811 11:50:54.691260 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:54.691263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:54.691325 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:54.691418 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:54.692988 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.693582 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.694754 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.694978 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.695071 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:54.698782 (+ 3711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.66:48683: BlockingRecv error: recv got EOF from 127.3.221.66:48683 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 11:50:54.699468 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:54.688745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.66:48683)
0811 11:50:54.689294 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:54.689349 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:54.690967 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:54.691637 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:54.691647 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:54.692153 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:54.692820 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:54.692837 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.693719 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.693723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:54.694621 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:54.694631 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.695136 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.695140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:54.695343 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:54.695938 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:54.695956 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:54.698578 (+ 2622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":129,"threads_started":1}
W20260811 11:50:54.700085 18960 heartbeater.cc:644] Failed to heartbeat to 127.3.221.126:40481 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.221.126:40481: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Aug 11 11:50:54 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:54 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:54.705467 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:54.700563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.66:52823)
0811 11:50:54.700722 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:54.700737 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:54.700820 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:54.701047 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:54.701049 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:54.701211 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:54.701338 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:54.701343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.702044 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.702047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:54.702585 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:54.702593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.703007 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.703011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:54.703096 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:54.703544 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:54.703559 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:54.705336 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97}
W20260811 11:50:54.705623 18740 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:54.700681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:52823 (local address 127.3.221.126:40481)
0811 11:50:54.700805 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:54.700809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:54.700823 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:50:54.700913 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:54.700916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:54.700961 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:54.701021 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:54.701444 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.701945 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.702691 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.702910 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.702995 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:54.705506 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.66:52823: BlockingRecv error: recv got EOF from 127.3.221.66:52823 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 11 11:50:54 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:54 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:54.710701 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:54.706056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.66:52523)
0811 11:50:54.706202 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:54.706216 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:54.706290 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:54.706518 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:54.706520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:54.706660 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:54.706729 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:54.706734 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.707420 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.707424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:54.707929 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:54.707936 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.708372 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.708375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:54.708448 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:54.708860 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:54.708873 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:54.710578 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88}
W20260811 11:50:54.710808 18740 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:54.706140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:52523 (local address 127.3.221.126:40481)
0811 11:50:54.706249 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:54.706253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:54.706266 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:50:54.706387 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:54.706389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:54.706432 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:54.706491 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:54.706831 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.707317 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.708049 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.708276 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.708360 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:54.710708 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.66:52523: BlockingRecv error: recv got EOF from 127.3.221.66:52523 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260811 11:50:54.710933 18960 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 11:50:54.810559 18967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:50:54.810827 18967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:50:54.810889 18967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:50:54.815016 18967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:50:54.815091 18967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:50:54.815172 18967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:50:54.820729 18967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/krb5kdc/oryx_127.3.221.67.keytab
--principal=oryx/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:33277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=46425
--tserver_master_addrs=127.3.221.126:40481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:50:54.821965 18967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:50:54.823015 18967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:50:54.833089 18975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:50:54.833168 18981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:54.833158 18967 server_base.cc:1034] running on GCE node
W20260811 11:50:54.833082 18974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:50:54.833778 18967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:50:54.834446 18967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:50:54.835608 18967 hybrid_clock.cc:648] HybridClock initialized: now 1786449054835578 us; error 41 us; skew 500 ppm
Aug 11 11:50:54 dist-test-slave-bndk krb5kdc[17490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449054, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:50:54.838651 18967 init.cc:377] Logged in from keytab as oryx/127.3.221.67@KRBTEST.COM (short username oryx)
I20260811 11:50:54.839573 18967 webserver.cc:460] Webserver started at http://127.3.221.67:46425/ using document root <none> and password file <none>
I20260811 11:50:54.840149 18967 fs_manager.cc:359] Metadata directory not provided
I20260811 11:50:54.840231 18967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:50:54.844291 18967 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260811 11:50:54.846519 18993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:54.847887 18967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 11:50:54.848018 18967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "fe1e0590b2824091a71eb7aec531ddcd"
format_stamp: "Formatted at 2026-08-11 11:50:53 on dist-test-slave-bndk"
server_key: "6c8f3681df833072fbb49fd558ab902b"
server_key_iv: "03fe3cdfd76fe12a14e91fc06e7fcca2"
server_key_version: "encryptionkey@0"
I20260811 11:50:54.848582 18967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:50:54.861114 18967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:50:54.864302 18967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:50:54.864485 18967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:50:54.865115 18967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:50:54.866011 18967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:50:54.866060 18967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:54.866101 18967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:50:54.866127 18967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:50:54.875403 18967 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:33277
I20260811 11:50:54.875453 19161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:33277 every 8 connection(s)
I20260811 11:50:54.876485 18967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:50:54.878515 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 18967
Aug 11 11:50:54 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:54 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:54.888537 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:54.879508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34622 (local address 127.3.221.126:40481)
0811 11:50:54.879728 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:54.879732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:54.879746 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:50:54.879795 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:54.879798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:54.879844 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:54.879925 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:54.880862 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.881536 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.882693 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.882992 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.883088 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:54.884704 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:50:54.884709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:50:54.884724 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:50:54.884754 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:50:54.886427 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:54.886908 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:54.886914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:54.886919 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:54.887061 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:50:54.887349 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:50:54.887352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:50:54.887354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:50:54.887710 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:50:54.887888 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0811 11:50:54.888142 (+ 254us) server_negotiation.cc:299] Negotiation successful
0811 11:50:54.888361 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":158,"thread_start_us":72,"threads_started":1}
W20260811 11:50:54.888831 18740 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:54.878422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:45105 (local address 127.3.221.126:40481)
0811 11:50:54.878576 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:54.878581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:54.879551 (+ 970us) server_negotiation.cc:407] Connection header received
0811 11:50:54.881195 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:54.881198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:54.881280 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:54.881357 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:54.882851 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.883497 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.885016 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.885307 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.885414 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:54.888736 (+ 3322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.67:45105: BlockingRecv error: recv got EOF from 127.3.221.67:45105 (error 108)
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":55}
I20260811 11:50:54.889549 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:54.878845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.67:45105)
0811 11:50:54.879384 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:54.879447 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:54.880991 (+ 1544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:54.881529 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:54.881536 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:54.882025 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:54.882720 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:54.882739 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.883625 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.883628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:54.884891 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:54.884901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.885455 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.885458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:54.885632 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:54.886124 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:54.886139 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:54.888610 (+ 2471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":129,"threads_started":1}
W20260811 11:50:54.890467 19162 heartbeater.cc:644] Failed to heartbeat to 127.3.221.126:40481 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.221.126:40481: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Aug 11 11:50:54 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:54 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:54.895701 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:54.890899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.67:43433)
0811 11:50:54.891018 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:54.891032 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:54.891105 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:54.891539 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:54.891541 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:54.891680 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:54.891773 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:54.891778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.892447 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.892449 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:54.893000 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:54.893007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.893485 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.893487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:54.893556 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:54.893939 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:54.893949 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:54.895573 (+ 1624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260811 11:50:54.895804 19167 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:54.890948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:43433 (local address 127.3.221.126:40481)
0811 11:50:54.891349 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:54.891353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:54.891365 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:50:54.891399 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:54.891402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:54.891442 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:54.891512 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:54.891857 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.892370 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.893096 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.893383 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.893463 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:54.895708 (+ 2245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.67:43433: BlockingRecv error: recv got EOF from 127.3.221.67:43433 (error 108)
Metrics: {"server-negotiator.queue_time_us":328}
Aug 11 11:50:54 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:54 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:54.900372 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:54.896239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.67:46967)
0811 11:50:54.896345 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:54.896359 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:54.896433 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:54.896620 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:54.896621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:54.896736 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:54.896802 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:54.896818 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.897485 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.897488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:54.897987 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:54.897993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.898329 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.898331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:54.898383 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:54.898802 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:54.898824 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:54.900275 (+ 1451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260811 11:50:54.900501 19167 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:54.896314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:46967 (local address 127.3.221.126:40481)
0811 11:50:54.896460 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:54.896465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:54.896476 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:50:54.896507 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:54.896509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:54.896540 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:54.896601 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:54.896890 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.897407 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.898066 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:54.898256 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:54.898320 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:54.900388 (+ 2068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.67:46967: BlockingRecv error: recv got EOF from 127.3.221.67:46967 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260811 11:50:54.900561 19162 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 11 11:50:55 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:55 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:55.528393 19168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:55.523085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.65:39109)
0811 11:50:55.523394 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:55.523412 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:55.523511 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:55.523781 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:55.523784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:55.523990 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:55.524118 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:55.524131 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:55.524843 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:55.524846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:55.525461 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:55.525472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:55.525857 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:55.525859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:55.525917 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:55.526453 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:55.526469 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:55.528263 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":78,"threads_started":1}
W20260811 11:50:55.528558 19169 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:55.523190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:39109 (local address 127.3.221.126:40481)
0811 11:50:55.523483 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:55.523487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:55.523500 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:50:55.523587 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:55.523589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:55.523632 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:55.523706 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:55.524221 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:55.524746 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:55.525556 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:55.525770 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:55.525844 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:55.528415 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.65:39109: BlockingRecv error: recv got EOF from 127.3.221.65:39109 (error 108)
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":100,"threads_started":1}
Aug 11 11:50:55 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:55 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:55.717664 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:55.711710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.66:36645)
0811 11:50:55.712028 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:55.712044 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:55.712192 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:55.712562 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:55.712566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:55.712777 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:55.712901 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:55.712911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:55.713896 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:55.713899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:55.714485 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:55.714494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:55.714995 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:55.714998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:55.715069 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:55.715545 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:55.715557 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:55.717486 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":103,"threads_started":1}
W20260811 11:50:55.717756 19169 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:55.711857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:36645 (local address 127.3.221.126:40481)
0811 11:50:55.712017 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:55.712022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:55.712123 (+ 101us) server_negotiation.cc:407] Connection header received
0811 11:50:55.712342 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:55.712345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:55.712401 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:55.712491 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:55.713076 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:55.713769 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:55.714621 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:55.714863 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:55.714954 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:55.717649 (+ 2695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.66:36645: BlockingRecv error: recv got EOF from 127.3.221.66:36645 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 11 11:50:55 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:55 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:55.907440 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:55.901268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.67:48569)
0811 11:50:55.901586 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:55.901602 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:55.901728 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:55.901985 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:55.901988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:55.902197 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:55.902295 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:55.902301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:55.903360 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:55.903365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:55.904155 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:55.904169 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:55.904738 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:55.904741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:55.904832 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:55.905392 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:55.905404 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:55.907269 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":91,"threads_started":1}
W20260811 11:50:55.907565 19169 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:55.901376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:48569 (local address 127.3.221.126:40481)
0811 11:50:55.901555 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:55.901560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:55.901674 (+ 114us) server_negotiation.cc:407] Connection header received
0811 11:50:55.901819 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:55.901822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:55.901872 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:55.901941 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:55.902420 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:55.903228 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:55.904286 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:55.904609 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:55.904714 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:55.907425 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.67:48569: BlockingRecv error: recv got EOF from 127.3.221.67:48569 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 11 11:50:56 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:56 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:56.534946 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:56.529201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.65:41193)
0811 11:50:56.529512 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:56.529528 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:56.529643 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:56.529914 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:56.529917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:56.530122 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:56.530224 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:56.530230 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:56.531007 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:56.531010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:56.531584 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:56.531593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:56.532051 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:56.532054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:56.532138 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:56.532641 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:56.532653 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:56.534773 (+ 2120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":93,"threads_started":1}
W20260811 11:50:56.535111 19173 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:56.529302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:41193 (local address 127.3.221.126:40481)
0811 11:50:56.529598 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:56.529604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:56.529617 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:50:56.529741 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:56.529745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:56.529794 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:56.529868 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:56.530351 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:56.530893 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:56.531706 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:56.531941 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:56.532029 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:56.534956 (+ 2927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.65:41193: BlockingRecv error: recv got EOF from 127.3.221.65:41193 (error 108)
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":112,"threads_started":1}
Aug 11 11:50:56 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:56 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:56.724576 19175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:56.718673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.66:45965)
0811 11:50:56.718975 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:56.718989 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:56.719098 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:56.719444 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:56.719447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:56.719641 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:56.719747 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:56.719753 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:56.720535 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:56.720538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:56.721099 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:56.721106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:56.721685 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:56.721689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:56.721765 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:56.722424 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:56.722441 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:56.724383 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":93,"threads_started":1}
W20260811 11:50:56.724653 19173 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:56.718850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:45965 (local address 127.3.221.126:40481)
0811 11:50:56.718984 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:56.718988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:56.719058 (+ 70us) server_negotiation.cc:407] Connection header received
0811 11:50:56.719223 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:56.719226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:56.719281 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:56.719381 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:56.719887 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:56.720442 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:56.721179 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:56.721543 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:56.721630 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:56.724548 (+ 2918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.66:45965: BlockingRecv error: recv got EOF from 127.3.221.66:45965 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 11 11:50:56 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:56 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:56.913702 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:56.908234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.67:55827)
0811 11:50:56.908500 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:56.908518 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:56.908631 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:56.908886 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:56.908889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:56.909086 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:56.909180 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:56.909186 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:56.909983 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:56.909986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:56.910526 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:56.910534 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:56.910958 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:56.910962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:56.911042 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:56.911517 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:56.911531 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:56.913554 (+ 2023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":81,"threads_started":1}
W20260811 11:50:56.913784 19173 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:56.908420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:55827 (local address 127.3.221.126:40481)
0811 11:50:56.908556 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:56.908561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:56.908575 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:50:56.908726 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:56.908729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:56.908777 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:56.908845 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:56.909345 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:56.909873 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:56.910631 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:56.910854 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:56.910942 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:56.913686 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.67:55827: BlockingRecv error: recv got EOF from 127.3.221.67:55827 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 11 11:50:57 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:57 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:57.540934 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:57.535693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.65:44111)
0811 11:50:57.535998 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:57.536061 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:57.536180 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:57.536439 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:57.536441 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:57.536647 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:57.536749 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:57.536755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:57.537579 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:57.537581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:57.538144 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:57.538153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:57.538541 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:57.538543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:57.538603 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:57.539083 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:57.539094 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:57.540813 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":105,"threads_started":1}
W20260811 11:50:57.541082 19180 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:57.535762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:44111 (local address 127.3.221.126:40481)
0811 11:50:57.536009 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:57.536013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:57.536127 (+ 114us) server_negotiation.cc:407] Connection header received
0811 11:50:57.536263 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:57.536265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:57.536335 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:57.536412 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:57.536851 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:57.537484 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:57.538232 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:57.538461 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:57.538532 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:57.540960 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.65:44111: BlockingRecv error: recv got EOF from 127.3.221.65:44111 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":65,"threads_started":1}
Aug 11 11:50:57 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:57 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:57.731124 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:57.725491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.66:35221)
0811 11:50:57.725960 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:57.725975 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:57.726098 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:57.726334 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:57.726337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:57.726549 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:57.726646 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:57.726653 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:57.727412 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:57.727415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:57.727979 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:57.727987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:57.728426 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:57.728429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:57.728511 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:57.729032 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:57.729045 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:57.730995 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":97,"threads_started":1}
W20260811 11:50:57.731300 19180 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:57.725583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:35221 (local address 127.3.221.126:40481)
0811 11:50:57.725731 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:57.725736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:57.726058 (+ 322us) server_negotiation.cc:407] Connection header received
0811 11:50:57.726179 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:57.726183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:57.726231 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:57.726295 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:57.726767 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:57.727299 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:57.728096 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:57.728318 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:57.728408 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:57.731152 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.66:35221: BlockingRecv error: recv got EOF from 127.3.221.66:35221 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 11 11:50:57 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:57 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:57.920406 19182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:57.914599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.67:55059)
0811 11:50:57.914914 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:57.914931 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:57.915054 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:57.915304 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:57.915307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:57.915539 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:57.915656 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:57.915665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:57.916552 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:57.916555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:57.917111 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:57.917119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:57.917609 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:57.917613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:57.917681 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:57.918161 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:57.918179 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:57.920267 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":96,"threads_started":1}
W20260811 11:50:57.920585 19180 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:57.914744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:55059 (local address 127.3.221.126:40481)
0811 11:50:57.914887 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:57.914892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:57.915023 (+ 131us) server_negotiation.cc:407] Connection header received
0811 11:50:57.915129 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:57.915132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:57.915192 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:57.915259 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:57.915792 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:57.916449 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:57.917204 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:57.917498 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:57.917577 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:57.920465 (+ 2888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.67:55059: BlockingRecv error: recv got EOF from 127.3.221.67:55059 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 11 11:50:58 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:58 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:58.549822 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:58.541777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.65:43891)
0811 11:50:58.542075 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:58.542089 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:58.542192 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:58.542439 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:58.542442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:58.542625 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:58.542726 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:58.542732 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:58.543508 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:58.543511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:58.544068 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:58.544076 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:58.544511 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:58.544513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:58.544574 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:58.547826 (+ 3252us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:58.547840 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:58.549698 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":105,"threads_started":1}
W20260811 11:50:58.549993 19186 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:58.541882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:43891 (local address 127.3.221.126:40481)
0811 11:50:58.542127 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:58.542130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:58.542146 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:50:58.542274 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:58.542276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:58.542322 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:58.542407 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:58.542814 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:58.543407 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:58.544175 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:58.544423 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:58.544501 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:58.549848 (+ 5347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.65:43891: BlockingRecv error: recv got EOF from 127.3.221.65:43891 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":64,"threads_started":1}
Aug 11 11:50:58 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:58 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:58.738090 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:58.732036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.66:51963)
0811 11:50:58.732383 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:58.732398 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:58.732517 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:58.732800 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:58.732803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:58.732994 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:58.733097 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:58.733103 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:58.733960 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:58.733963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:58.734613 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:58.734626 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:58.735162 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:58.735166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:58.735244 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:58.735824 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:58.735838 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:58.737938 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":92,"threads_started":1}
W20260811 11:50:58.738194 19186 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:58.732204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:51963 (local address 127.3.221.126:40481)
0811 11:50:58.732365 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:58.732369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:58.732471 (+ 102us) server_negotiation.cc:407] Connection header received
0811 11:50:58.732595 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:58.732598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:58.732646 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:58.732718 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:58.733242 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:58.733798 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:58.734757 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:58.735003 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:58.735110 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:58.738095 (+ 2985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.66:51963: BlockingRecv error: recv got EOF from 127.3.221.66:51963 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 11 11:50:58 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:58 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:58.926921 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:58.921403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.67:37693)
0811 11:50:58.921680 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:58.921694 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:58.921812 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:58.922038 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:58.922040 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:58.922226 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:58.922325 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:58.922331 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:58.923052 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:58.923055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:58.923619 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:58.923627 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:58.924064 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:58.924066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:58.924126 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:58.924680 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:58.924692 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:58.926797 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":113,"threads_started":1}
W20260811 11:50:58.927134 19186 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:58.921435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:37693 (local address 127.3.221.126:40481)
0811 11:50:58.921607 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:58.921612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:58.921763 (+ 151us) server_negotiation.cc:407] Connection header received
0811 11:50:58.921889 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:58.921891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:58.921940 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:58.922014 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:58.922430 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:58.922945 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:58.923741 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:58.923970 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:58.924048 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:58.927014 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.67:37693: BlockingRecv error: recv got EOF from 127.3.221.67:37693 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 11 11:50:59 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:59 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:59.556598 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:59.550665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.65:49667)
0811 11:50:59.550955 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:59.550969 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:59.551060 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:59.551298 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:59.551300 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:59.551504 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:59.551603 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:59.551632 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:59.552433 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:59.552436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:59.553044 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:59.553054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:59.553699 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:59.553703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:59.553792 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:59.554371 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:59.554386 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:59.556444 (+ 2058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":96,"threads_started":1}
W20260811 11:50:59.556707 19190 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:59.550760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:49667 (local address 127.3.221.126:40481)
0811 11:50:59.551007 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:59.551011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:59.551025 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:50:59.551138 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:59.551140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:59.551187 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:59.551275 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:59.551758 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:59.552331 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:59.553149 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:59.553454 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:59.553558 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:59.556591 (+ 3033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.65:49667: BlockingRecv error: recv got EOF from 127.3.221.65:49667 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":63,"threads_started":1}
Aug 11 11:50:59 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:59 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:59.744617 19194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:59.739085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.66:57841)
0811 11:50:59.739401 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:59.739428 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:59.739544 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:59.739795 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:59.739798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:59.740008 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:59.740102 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:59.740108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:59.740874 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:59.740877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:59.741498 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:59.741509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:59.741943 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:59.741947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:59.742026 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:59.742532 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:59.742544 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:59.744498 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":112,"threads_started":1}
W20260811 11:50:59.744762 19190 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:59.739248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:57841 (local address 127.3.221.126:40481)
0811 11:50:59.739409 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:59.739414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:59.739500 (+ 86us) server_negotiation.cc:407] Connection header received
0811 11:50:59.739636 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:59.739639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:59.739688 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:59.739755 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:59.740222 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:59.740763 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:59.741612 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:59.741836 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:59.741935 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:59.744640 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.66:57841: BlockingRecv error: recv got EOF from 127.3.221.66:57841 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 11 11:50:59 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:50:59 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:50:59.933569 19195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:50:59.927777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.67:42497)
0811 11:50:59.928081 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0811 11:50:59.928096 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:50:59.928205 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:50:59.928597 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:50:59.928599 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:50:59.928802 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:50:59.928903 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:59.928909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:59.929810 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:59.929813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:59.930386 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:50:59.930394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:50:59.930876 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:50:59.930880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:50:59.930953 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:59.931449 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:50:59.931460 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:50:59.933436 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":100,"threads_started":1}
W20260811 11:50:59.933754 19190 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:50:59.927907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:42497 (local address 127.3.221.126:40481)
0811 11:50:59.928095 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0811 11:50:59.928099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:50:59.928155 (+ 56us) server_negotiation.cc:407] Connection header received
0811 11:50:59.928383 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:50:59.928385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:50:59.928438 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:50:59.928515 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:50:59.929062 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:59.929675 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:59.930510 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:50:59.930743 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:50:59.930826 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:50:59.933593 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.67:42497: BlockingRecv error: recv got EOF from 127.3.221.67:42497 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 11 11:51:00 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:00 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:00.562741 19196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:00.557511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.65:33427)
0811 11:51:00.557799 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:00.557813 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:00.557910 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:00.558134 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:00.558136 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:00.558325 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:00.558450 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:00.558456 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:00.559159 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:00.559161 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:00.559736 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:00.559743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:00.560121 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:00.560123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:00.560182 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:00.560680 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:00.560691 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:00.562584 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":80,"threads_started":1}
W20260811 11:51:00.562860 19197 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:00.557616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:33427 (local address 127.3.221.126:40481)
0811 11:51:00.557877 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:00.557880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:00.557895 (+ 15us) server_negotiation.cc:407] Connection header received
0811 11:51:00.557982 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:00.557984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:00.558027 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:00.558106 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:00.558547 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:00.559078 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:00.559823 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:00.560038 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:00.560108 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:00.562742 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.65:33427: BlockingRecv error: recv got EOF from 127.3.221.65:33427 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":94,"threads_started":1}
Aug 11 11:51:00 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:00 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:00.750841 19200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:00.745426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.66:59371)
0811 11:51:00.745768 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:00.745784 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:00.745898 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:00.746129 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:00.746132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:00.746332 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:00.746435 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:00.746440 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:00.747197 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:00.747200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:00.747772 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:00.747781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:00.748224 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:00.748227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:00.748294 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:00.748780 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:00.748799 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:00.750682 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":101,"threads_started":1}
W20260811 11:51:00.750969 19197 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:00.745543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:59371 (local address 127.3.221.126:40481)
0811 11:51:00.745711 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:00.745715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:00.745852 (+ 137us) server_negotiation.cc:407] Connection header received
0811 11:51:00.745974 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:00.745976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:00.746027 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:00.746097 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:00.746551 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:00.747098 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:00.747896 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:00.748130 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:00.748209 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:00.750847 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.66:59371: BlockingRecv error: recv got EOF from 127.3.221.66:59371 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 11 11:51:00 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:00 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:00.940086 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:00.934499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.67:54141)
0811 11:51:00.934830 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:00.934845 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:00.934960 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:00.935359 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:00.935361 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:00.935591 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:00.935695 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:00.935702 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:00.936521 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:00.936524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:00.937111 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:00.937121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:00.937571 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:00.937575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:00.937645 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:00.938115 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:00.938126 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:00.939928 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":106,"threads_started":1}
W20260811 11:51:00.940187 19197 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:00.934607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:54141 (local address 127.3.221.126:40481)
0811 11:51:00.934772 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:00.934777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:00.934916 (+ 139us) server_negotiation.cc:407] Connection header received
0811 11:51:00.935114 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:00.935116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:00.935170 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:00.935323 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:00.935816 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:00.936360 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:00.937210 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:00.937458 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:00.937551 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:00.940083 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.67:54141: BlockingRecv error: recv got EOF from 127.3.221.67:54141 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 11 11:51:01 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:01 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:01.569195 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:01.563646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.65:55263)
0811 11:51:01.563980 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:01.563993 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:01.564100 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:01.564401 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:01.564403 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:01.564591 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:01.564690 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:01.564695 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:01.565558 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:01.565561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:01.566118 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:01.566127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:01.566597 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:01.566599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:01.566660 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:01.567159 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:01.567170 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:01.569073 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":77,"threads_started":1}
W20260811 11:51:01.569403 19203 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:01.563762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:55263 (local address 127.3.221.126:40481)
0811 11:51:01.563992 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:01.563996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:01.564091 (+ 95us) server_negotiation.cc:407] Connection header received
0811 11:51:01.564215 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:01.564219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:01.564268 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:01.564356 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:01.564776 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:01.565434 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:01.566253 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:01.566490 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:01.566576 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:01.569265 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.65:55263: BlockingRecv error: recv got EOF from 127.3.221.65:55263 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
Aug 11 11:51:01 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:01 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:01.756974 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:01.751638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.66:45221)
0811 11:51:01.751946 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:01.751960 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:01.752070 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:01.752327 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:01.752329 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:01.752526 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:01.752627 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:01.752635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:01.753481 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:01.753484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:01.754045 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:01.754053 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:01.754558 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:01.754560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:01.754619 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:01.755084 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:01.755095 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:01.756850 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":76,"threads_started":1}
W20260811 11:51:01.757129 19203 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:01.751750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:45221 (local address 127.3.221.126:40481)
0811 11:51:01.751951 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:01.751956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:01.752015 (+ 59us) server_negotiation.cc:407] Connection header received
0811 11:51:01.752145 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:01.752147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:01.752222 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:01.752298 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:01.752785 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:01.753385 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:01.754178 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:01.754462 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:01.754553 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:01.757011 (+ 2458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.66:45221: BlockingRecv error: recv got EOF from 127.3.221.66:45221 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 11 11:51:01 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:01 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:01.946813 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:01.941054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.67:55143)
0811 11:51:01.941573 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:01.941587 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:01.941704 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:01.941981 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:01.941984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:01.942184 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:01.942276 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:01.942282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:01.943081 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:01.943084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:01.943671 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:01.943681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:01.944146 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:01.944149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:01.944209 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:01.944741 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:01.944753 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:01.946685 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":116,"threads_started":1}
W20260811 11:51:01.946954 19203 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:01.941133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:55143 (local address 127.3.221.126:40481)
0811 11:51:01.941318 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:01.941323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:01.941686 (+ 363us) server_negotiation.cc:407] Connection header received
0811 11:51:01.941790 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:01.941793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:01.941853 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:01.941932 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:01.942424 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:01.942960 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:01.943813 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:01.944046 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:01.944140 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:01.946842 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.67:55143: BlockingRecv error: recv got EOF from 127.3.221.67:55143 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 11 11:51:02 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:02 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:02.575640 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:02.570058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.65:53913)
0811 11:51:02.570359 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:02.570374 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:02.570477 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:02.570781 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:02.570784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:02.570984 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:02.571090 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:02.571096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:02.571909 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:02.571913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:02.572613 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:02.572626 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:02.573087 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:02.573090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:02.573177 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:02.573721 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:02.573736 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:02.575477 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":84,"threads_started":1}
W20260811 11:51:02.575764 19209 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:02.570270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:53913 (local address 127.3.221.126:40481)
0811 11:51:02.570497 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:02.570501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:02.570532 (+ 31us) server_negotiation.cc:407] Connection header received
0811 11:51:02.570574 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:02.570576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:02.570658 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:02.570734 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:02.571187 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:02.571773 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:02.572734 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:02.572970 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:02.573062 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:02.575641 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.65:53913: BlockingRecv error: recv got EOF from 127.3.221.65:53913 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":68,"threads_started":1}
Aug 11 11:51:02 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:02 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:02.763789 19210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:02.757854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.66:44577)
0811 11:51:02.758137 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:02.758151 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:02.758261 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:02.758523 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:02.758525 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:02.758715 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:02.758819 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:02.758838 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:02.759793 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:02.759796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:02.760368 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:02.760377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:02.760915 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:02.760919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:02.761095 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:02.761695 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:02.761710 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:02.763670 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":81,"threads_started":1}
W20260811 11:51:02.763921 19209 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:02.757988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:44577 (local address 127.3.221.126:40481)
0811 11:51:02.758131 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:02.758135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:02.758224 (+ 89us) server_negotiation.cc:407] Connection header received
0811 11:51:02.758360 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:02.758362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:02.758410 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:02.758476 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:02.758969 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:02.759669 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:02.760496 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:02.760755 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:02.760848 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:02.763813 (+ 2965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.66:44577: BlockingRecv error: recv got EOF from 127.3.221.66:44577 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 11 11:51:02 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:02 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:02.953259 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:02.947676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.67:56003)
0811 11:51:02.947949 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:02.947963 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:02.948075 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:02.948305 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:02.948307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:02.948524 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:02.948624 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:02.948630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:02.949465 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:02.949468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:02.950040 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:02.950051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:02.950458 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:02.950460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:02.950524 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:02.951050 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:02.951061 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:02.953094 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
W20260811 11:51:02.953478 19209 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:02.947786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:56003 (local address 127.3.221.126:40481)
0811 11:51:02.947950 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:02.947954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:02.948067 (+ 113us) server_negotiation.cc:407] Connection header received
0811 11:51:02.948146 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:02.948149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:02.948202 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:02.948285 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:02.948763 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:02.949335 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:02.950133 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:02.950363 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:02.950432 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:02.953347 (+ 2915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.67:56003: BlockingRecv error: recv got EOF from 127.3.221.67:56003 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 11 11:51:03 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:03 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:03.586643 19212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:03.576606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.65:38129)
0811 11:51:03.576913 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:03.576931 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:03.577058 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:03.577393 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:03.577396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:03.577651 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:03.577781 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:03.577789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:03.578664 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:03.578668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:03.579230 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:03.579239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:03.579754 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:03.579757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:03.579831 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:03.584390 (+ 4559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:03.584415 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:03.586494 (+ 2079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":107,"threads_started":1}
W20260811 11:51:03.586872 19214 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:03.576732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:38129 (local address 127.3.221.126:40481)
0811 11:51:03.576985 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:03.576990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:03.577006 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:51:03.577158 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:03.577161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:03.577247 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:03.577347 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:03.577934 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:03.578536 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:03.579384 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:03.579618 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:03.579700 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:03.586699 (+ 6999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.65:38129: BlockingRecv error: recv got EOF from 127.3.221.65:38129 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":67,"threads_started":1}
Aug 11 11:51:03 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:03 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:03.770437 19215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:03.764684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.66:55633)
0811 11:51:03.765028 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:03.765045 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:03.765175 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:03.765555 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:03.765558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:03.765756 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:03.765860 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:03.765867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:03.766668 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:03.766672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:03.767230 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:03.767239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:03.767685 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:03.767688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:03.767769 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:03.768276 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:03.768289 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:03.770290 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":138,"threads_started":1}
W20260811 11:51:03.770613 19214 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:03.764878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:55633 (local address 127.3.221.126:40481)
0811 11:51:03.765031 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:03.765036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:03.765119 (+ 83us) server_negotiation.cc:407] Connection header received
0811 11:51:03.765349 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:03.765353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:03.765412 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:03.765495 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:03.765986 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:03.766549 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:03.767354 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:03.767571 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:03.767657 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:03.770477 (+ 2820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.66:55633: BlockingRecv error: recv got EOF from 127.3.221.66:55633 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 11 11:51:03 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:03 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:03.959853 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:03.954173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.67:60031)
0811 11:51:03.954644 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:03.954657 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:03.954797 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:03.955070 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:03.955073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:03.955280 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:03.955384 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:03.955392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:03.956163 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:03.956166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:03.956739 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:03.956749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:03.957273 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:03.957278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:03.957364 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:03.957896 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:03.957908 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:03.959722 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":92,"threads_started":1}
W20260811 11:51:03.959964 19214 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:03.954310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:60031 (local address 127.3.221.126:40481)
0811 11:51:03.954455 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:03.954460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:03.954720 (+ 260us) server_negotiation.cc:407] Connection header received
0811 11:51:03.954902 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:03.954905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:03.954960 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:03.955027 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:03.955479 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:03.956049 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:03.956859 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:03.957101 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:03.957184 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:03.959858 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.67:60031: BlockingRecv error: recv got EOF from 127.3.221.67:60031 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 11 11:51:04 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:04 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:04.593417 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:04.587493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.65:48969)
0811 11:51:04.587814 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:04.587830 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:04.587938 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:04.588174 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:04.588175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:04.588364 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:04.588472 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:04.588478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:04.589206 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:04.589208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:04.589809 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:04.589820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:04.590415 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:04.590418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:04.590502 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:04.591086 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:04.591099 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:04.593249 (+ 2150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":96,"threads_started":1}
W20260811 11:51:04.593575 19221 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:04.587613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:48969 (local address 127.3.221.126:40481)
0811 11:51:04.587858 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:04.587862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:04.587878 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:51:04.588022 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:04.588024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:04.588069 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:04.588146 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:04.588568 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:04.589098 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:04.589950 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:04.590189 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:04.590316 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:04.593443 (+ 3127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.65:48969: BlockingRecv error: recv got EOF from 127.3.221.65:48969 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":60,"threads_started":1}
Aug 11 11:51:04 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:04 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:04.776908 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:04.771374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.66:43831)
0811 11:51:04.771734 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:04.771748 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:04.771863 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:04.772126 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:04.772129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:04.772340 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:04.772439 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:04.772446 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:04.773211 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:04.773245 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:04.773804 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:04.773813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:04.774268 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:04.774272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:04.774350 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:04.774847 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:04.774859 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:04.776785 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":131,"threads_started":1}
W20260811 11:51:04.777076 19221 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:04.771447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:43831 (local address 127.3.221.126:40481)
0811 11:51:04.771580 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:04.771585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:04.771835 (+ 250us) server_negotiation.cc:407] Connection header received
0811 11:51:04.771957 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:04.771960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:04.772017 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:04.772083 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:04.772557 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:04.773092 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:04.773927 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:04.774163 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:04.774253 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:04.776944 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.66:43831: BlockingRecv error: recv got EOF from 127.3.221.66:43831 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 11 11:51:04 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:04 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:04.966106 19232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:04.960730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.67:47191)
0811 11:51:04.961039 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:04.961054 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:04.961179 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:04.961499 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:04.961502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:04.961726 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:04.961859 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:04.961867 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:04.962643 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:04.962646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:04.963230 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:04.963238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:04.963627 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:04.963629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:04.963688 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:04.964186 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:04.964197 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:04.965949 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":92,"threads_started":1}
W20260811 11:51:04.966204 19221 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:04.960849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:47191 (local address 127.3.221.126:40481)
0811 11:51:04.960993 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:04.960998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:04.961143 (+ 145us) server_negotiation.cc:407] Connection header received
0811 11:51:04.961308 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:04.961311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:04.961371 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:04.961455 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:04.961988 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:04.962550 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:04.963316 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:04.963545 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:04.963616 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:04.966092 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.67:47191: BlockingRecv error: recv got EOF from 127.3.221.67:47191 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 11 11:51:05 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:05 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:05.599571 19234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:05.594309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.65:36889)
0811 11:51:05.594618 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:05.594632 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:05.594729 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:05.594960 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:05.594962 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:05.595147 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:05.595247 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:05.595252 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:05.595955 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:05.595958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:05.596521 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:05.596529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:05.596949 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:05.596951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:05.597008 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:05.597542 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:05.597555 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:05.599444 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":86,"threads_started":1}
W20260811 11:51:05.599750 19235 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:05.594439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:36889 (local address 127.3.221.126:40481)
0811 11:51:05.594701 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:05.594705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:05.594721 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:51:05.594802 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:05.594804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:05.594849 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:05.594935 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:05.595339 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:05.595871 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:05.596628 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:05.596861 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:05.596936 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:05.599612 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.65:36889: BlockingRecv error: recv got EOF from 127.3.221.65:36889 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
Aug 11 11:51:05 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:05 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:05.783689 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:05.777788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.66:41601)
0811 11:51:05.778076 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:05.778089 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:05.778203 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:05.778576 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:05.778580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:05.778829 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:05.778957 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:05.778966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:05.779959 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:05.779962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:05.780522 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:05.780531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:05.781029 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:05.781032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:05.781107 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:05.781644 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:05.781657 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:05.783561 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":92,"threads_started":1}
W20260811 11:51:05.783900 19235 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:05.777888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:41601 (local address 127.3.221.126:40481)
0811 11:51:05.778088 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:05.778092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:05.778169 (+ 77us) server_negotiation.cc:407] Connection header received
0811 11:51:05.778311 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:05.778314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:05.778381 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:05.778475 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:05.779091 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:05.779849 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:05.780665 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:05.780888 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:05.780977 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:05.783770 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.66:41601: BlockingRecv error: recv got EOF from 127.3.221.66:41601 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 11 11:51:05 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:05 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:05.972707 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:05.966954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.67:57417)
0811 11:51:05.967234 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:05.967247 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:05.967362 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:05.967634 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:05.967637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:05.967851 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:05.967949 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:05.967955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:05.968724 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:05.968728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:05.969344 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:05.969353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:05.969811 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:05.969815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:05.969898 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:05.970435 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:05.970446 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:05.972544 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":88,"threads_started":1}
W20260811 11:51:05.972803 19235 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:05.967016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:57417 (local address 127.3.221.126:40481)
0811 11:51:05.967149 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:05.967153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:05.967319 (+ 166us) server_negotiation.cc:407] Connection header received
0811 11:51:05.967453 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:05.967456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:05.967527 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:05.967594 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:05.968079 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:05.968616 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:05.969470 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:05.969701 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:05.969794 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:05.972697 (+ 2903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.67:57417: BlockingRecv error: recv got EOF from 127.3.221.67:57417 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 11 11:51:06 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:06 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:06.606487 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:06.600480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.65:54297)
0811 11:51:06.600798 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:06.600813 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:06.600914 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:06.601149 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:06.601151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:06.601380 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:06.601507 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:06.601513 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:06.602222 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:06.602224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:06.602785 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:06.602793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:06.603346 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:06.603350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:06.603435 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:06.604035 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:06.604051 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:06.606323 (+ 2272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":87,"threads_started":1}
W20260811 11:51:06.606681 19241 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:06.600609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:54297 (local address 127.3.221.126:40481)
0811 11:51:06.600867 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:06.600871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:06.600885 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:51:06.600996 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:06.600998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:06.601044 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:06.601120 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:06.601603 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:06.602136 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:06.602913 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:06.603155 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:06.603247 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:06.606520 (+ 3273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.65:54297: BlockingRecv error: recv got EOF from 127.3.221.65:54297 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
Aug 11 11:51:06 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:06 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:06.789928 19243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:06.784548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.66:45635)
0811 11:51:06.784833 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:06.784846 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:06.784956 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:06.785185 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:06.785187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:06.785399 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:06.785504 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:06.785509 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:06.786226 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:06.786228 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:06.786789 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:06.786797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:06.787209 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:06.787211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:06.787271 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:06.787794 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:06.787807 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:06.789790 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":92,"threads_started":1}
W20260811 11:51:06.790066 19241 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:06.784643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:45635 (local address 127.3.221.126:40481)
0811 11:51:06.784779 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:06.784783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:06.784908 (+ 125us) server_negotiation.cc:407] Connection header received
0811 11:51:06.785048 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:06.785050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:06.785097 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:06.785156 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:06.785597 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:06.786129 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:06.786881 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:06.787112 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:06.787196 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:06.789954 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.66:45635: BlockingRecv error: recv got EOF from 127.3.221.66:45635 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 11 11:51:06 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:06 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:06.979718 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:06.973851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.67:34341)
0811 11:51:06.974128 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:06.974143 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:06.974260 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:06.974514 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:06.974517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:06.974755 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:06.974881 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:06.974889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:06.975838 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:06.975842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:06.976659 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:06.976672 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:06.977164 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:06.977168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:06.977276 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:06.977839 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:06.977854 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:06.979561 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":91,"threads_started":1}
W20260811 11:51:06.979815 19241 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:06.973973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:34341 (local address 127.3.221.126:40481)
0811 11:51:06.974128 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:06.974133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:06.974201 (+ 68us) server_negotiation.cc:407] Connection header received
0811 11:51:06.974356 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:06.974359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:06.974409 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:06.974502 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:06.975020 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:06.975711 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:06.976801 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:06.977048 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:06.977137 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:06.979711 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.67:34341: BlockingRecv error: recv got EOF from 127.3.221.67:34341 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 11 11:51:07 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:07 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:07.613395 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:07.607810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.65:42619)
0811 11:51:07.608090 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:07.608104 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:07.608199 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:07.608540 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:07.608544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:07.608783 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:07.608913 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:07.608920 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:07.609830 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:07.609834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:07.610419 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:07.610429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:07.610881 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:07.610884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:07.610971 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:07.611474 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:07.611485 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:07.613167 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":74,"threads_started":1}
W20260811 11:51:07.613478 19247 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:07.607981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:42619 (local address 127.3.221.126:40481)
0811 11:51:07.608279 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:07.608284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:07.608301 (+ 17us) server_negotiation.cc:407] Connection header received
0811 11:51:07.608345 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:07.608347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:07.608392 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:07.608474 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:07.609032 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:07.609720 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:07.610540 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:07.610774 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:07.610863 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:07.613359 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.65:42619: BlockingRecv error: recv got EOF from 127.3.221.65:42619 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":79,"threads_started":1}
Aug 11 11:51:07 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:07 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:07.796576 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:07.790875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.66:53263)
0811 11:51:07.791172 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:07.791186 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:07.791303 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:07.791554 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:07.791556 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:07.791766 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:07.791868 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:07.791874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:07.792680 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:07.792684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:07.793387 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:07.793400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:07.793909 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:07.793913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:07.793982 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:07.794549 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:07.794562 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:07.796440 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":113,"threads_started":1}
W20260811 11:51:07.796784 19247 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:07.791060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:53263 (local address 127.3.221.126:40481)
0811 11:51:07.791229 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:07.791234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:07.791253 (+ 19us) server_negotiation.cc:407] Connection header received
0811 11:51:07.791389 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:07.791391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:07.791441 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:07.791515 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:07.791982 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:07.792524 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:07.793527 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:07.793780 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:07.793865 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:07.796632 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.66:53263: BlockingRecv error: recv got EOF from 127.3.221.66:53263 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 11 11:51:07 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:07 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:07.985867 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:07.980681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.67:43031)
0811 11:51:07.981040 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:07.981056 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:07.981181 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:07.981501 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:07.981504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:07.981697 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:07.981799 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:07.981805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:07.982511 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:07.982513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:07.983061 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:07.983068 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:07.983431 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:07.983433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:07.983491 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:07.983935 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:07.983946 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:07.985746 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":80,"threads_started":1}
W20260811 11:51:07.986007 19247 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:07.980738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:43031 (local address 127.3.221.126:40481)
0811 11:51:07.980866 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:07.980871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:07.981121 (+ 250us) server_negotiation.cc:407] Connection header received
0811 11:51:07.981317 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:07.981321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:07.981384 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:07.981472 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:07.981895 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:07.982426 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:07.983144 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:07.983354 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:07.983422 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:07.985886 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.67:43031: BlockingRecv error: recv got EOF from 127.3.221.67:43031 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 11 11:51:08 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:08 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:08.624909 19250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:08.614294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.65:46953)
0811 11:51:08.614581 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:08.614596 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:08.614691 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:08.615031 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:08.615034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:08.615234 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:08.615341 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:08.615347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:08.616073 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:08.616075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:08.616708 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:08.616719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:08.617298 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:08.617301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:08.617384 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:08.622712 (+ 5328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:08.622728 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:08.624754 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":88,"threads_started":1}
W20260811 11:51:08.625079 19251 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:08.614412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:46953 (local address 127.3.221.126:40481)
0811 11:51:08.614770 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:08.614774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:08.614788 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:51:08.614829 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:08.614831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:08.614873 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:08.614967 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:08.615439 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:08.615980 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:08.616833 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:08.617150 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:08.617264 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:08.624928 (+ 7664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.65:46953: BlockingRecv error: recv got EOF from 127.3.221.65:46953 (error 108)
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":98,"threads_started":1}
Aug 11 11:51:08 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:08 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:08.802497 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:08.797374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.66:45655)
0811 11:51:08.797666 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:08.797680 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:08.797780 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:08.798007 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:08.798009 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:08.798203 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:08.798302 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:08.798308 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:08.799007 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:08.799010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:08.799600 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:08.799608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:08.799985 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:08.799988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:08.800047 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:08.800528 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:08.800539 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:08.802358 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":101,"threads_started":1}
W20260811 11:51:08.802609 19251 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:08.797508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:45655 (local address 127.3.221.126:40481)
0811 11:51:08.797641 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:08.797645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:08.797740 (+ 95us) server_negotiation.cc:407] Connection header received
0811 11:51:08.797851 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:08.797853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:08.797897 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:08.797981 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:08.798393 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:08.798928 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:08.799684 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:08.799896 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:08.799972 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:08.802504 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.66:45655: BlockingRecv error: recv got EOF from 127.3.221.66:45655 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 11 11:51:08 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:08 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.67@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:08.991901 19254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:08.986612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.67:35461)
0811 11:51:08.986911 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:08.986923 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:08.987022 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:08.987264 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:08.987266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:08.987448 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:08.987545 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:08.987551 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:08.988235 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:08.988237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:08.988828 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:08.988836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:08.989314 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:08.989318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:08.989389 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:08.989836 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:08.989847 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:08.991743 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":105,"threads_started":1}
W20260811 11:51:08.992010 19251 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:08.986664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:35461 (local address 127.3.221.126:40481)
0811 11:51:08.986779 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:08.986783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:08.986980 (+ 197us) server_negotiation.cc:407] Connection header received
0811 11:51:08.987084 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:08.987086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:08.987134 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:08.987201 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:08.987632 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:08.988153 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:08.988924 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:08.989136 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:08.989210 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:08.991894 (+ 2684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.67:35461: BlockingRecv error: recv got EOF from 127.3.221.67:35461 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 11 11:51:09 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:09 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.65@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:09.631273 19259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:09.625883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.65:57911)
0811 11:51:09.626171 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:09.626185 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:09.626277 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:09.626526 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:09.626528 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:09.626723 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:09.626831 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:09.626836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:09.627643 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:09.627647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:09.628269 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:09.628280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:09.628733 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:09.628735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:09.628797 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:09.629301 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:09.629312 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:09.631124 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":92,"threads_started":1}
W20260811 11:51:09.631417 19260 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:09.625991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:57911 (local address 127.3.221.126:40481)
0811 11:51:09.626240 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:09.626244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:09.626258 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:51:09.626360 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:09.626362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:09.626405 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:09.626511 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:09.626965 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:09.627542 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:09.628388 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:09.628642 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:09.628718 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:09.631277 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.65:57911: BlockingRecv error: recv got EOF from 127.3.221.65:57911 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":63,"threads_started":1}
Aug 11 11:51:09 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 11:51:09 dist-test-slave-bndk krb5kdc[17490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.221.66@KRBTEST.COM for oryx/127.3.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 11:51:09.808976 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:09.803455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40481 (local address 127.3.221.66:47315)
0811 11:51:09.803721 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:09.803735 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:09.803833 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:09.804054 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:09.804056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:09.804257 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:09.804364 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:09.804370 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:09.805168 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:09.805171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:09.805793 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:09.805804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:09.806312 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:09.806316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:09.806389 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:09.806901 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:09.806915 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:09.808844 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.221.126:40481: Server oryx/127.3.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
W20260811 11:51:09.809163 19260 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:09.803561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:47315 (local address 127.3.221.126:40481)
0811 11:51:09.803724 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:09.803728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:09.803805 (+ 77us) server_negotiation.cc:407] Connection header received
0811 11:51:09.803904 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:09.803906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:09.803952 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:09.804029 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:09.804488 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:09.805050 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:09.805900 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:09.806126 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:09.806206 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:09.809041 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.221.66:47315: BlockingRecv error: recv got EOF from 127.3.221.66:47315 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260811 11:51:09.884404 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 18485
I20260811 11:51:09.903944 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 18742
I20260811 11:51:09.921641 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 18967
I20260811 11:51:09.943327 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 18357
2026-08-11T11:51:09Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18292 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-11T11:51:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:51:09Z Disabled control of system clock
I20260811 11:51:09.980568 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:42889
--webserver_interface=127.3.221.126
--webserver_port=0
--builtin_ntp_servers=127.3.221.84:34731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:42889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260811 11:51:10.100961 19271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:51:10.101295 19271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:51:10.101332 19271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:51:10.105487 19271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:51:10.105561 19271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:51:10.105576 19271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:51:10.105592 19271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:51:10.105608 19271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:51:10.110628 19271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:42889
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:42889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:51:10.111768 19271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:51:10.113200 19271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:51:10.122592 19277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:51:10.122653 19280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:51:10.122613 19271 server_base.cc:1034] running on GCE node
W20260811 11:51:10.122611 19278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:51:10.123466 19271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:51:10.124722 19271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:51:10.125923 19271 hybrid_clock.cc:648] HybridClock initialized: now 1786449070125901 us; error 39 us; skew 500 ppm
I20260811 11:51:10.128199 19271 webserver.cc:460] Webserver started at http://127.3.221.126:34033/ using document root <none> and password file <none>
I20260811 11:51:10.128813 19271 fs_manager.cc:359] Metadata directory not provided
I20260811 11:51:10.128885 19271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:51:10.129101 19271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:51:10.131448 19271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "c76ae57301c64ea2856e474d1df8b874"
format_stamp: "Formatted at 2026-08-11 11:51:10 on dist-test-slave-bndk"
server_key: "2fbcc82f82d21383f0c201f8edd0c887"
server_key_iv: "c5d41c10e86cb852c9a5674c5757869b"
server_key_version: "encryptionkey@0"
I20260811 11:51:10.132064 19271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "c76ae57301c64ea2856e474d1df8b874"
format_stamp: "Formatted at 2026-08-11 11:51:10 on dist-test-slave-bndk"
server_key: "2fbcc82f82d21383f0c201f8edd0c887"
server_key_iv: "c5d41c10e86cb852c9a5674c5757869b"
server_key_version: "encryptionkey@0"
I20260811 11:51:10.136132 19271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 11:51:10.138999 19294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:10.140403 19271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 11:51:10.140545 19271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "c76ae57301c64ea2856e474d1df8b874"
format_stamp: "Formatted at 2026-08-11 11:51:10 on dist-test-slave-bndk"
server_key: "2fbcc82f82d21383f0c201f8edd0c887"
server_key_iv: "c5d41c10e86cb852c9a5674c5757869b"
server_key_version: "encryptionkey@0"
I20260811 11:51:10.140679 19271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:51:10.155597 19271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:51:10.156384 19271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:51:10.156585 19271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:51:10.165907 19271 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:42889
I20260811 11:51:10.165923 19376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:42889 every 8 connection(s)
I20260811 11:51:10.166976 19271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:51:10.167574 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 19271
I20260811 11:51:10.167716 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:51:10.168071 3957 external_mini_cluster.cc:1428] Setting key 0596e205a8f839a9dae82bd2c7fae2ad
I20260811 11:51:10.170475 19377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:51:10.178045 19377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874: Bootstrap starting.
I20260811 11:51:10.179203 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:10.169632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33826 (local address 127.3.221.126:42889)
0811 11:51:10.170393 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:10.170425 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:10.170472 (+ 47us) server_negotiation.cc:407] Connection header received
0811 11:51:10.171378 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:10.171402 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:10.171922 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:10.172316 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:10.172766 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:10.174260 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:10.175198 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:10.175889 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:10.176056 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:10.176961 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:51:10.176968 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:51:10.176985 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:51:10.177560 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:51:10.177641 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:51:10.177670 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:51:10.177771 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 11:51:10.177949 (+ 178us) server_negotiation.cc:299] Negotiation successful
0811 11:51:10.178242 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":145,"threads_started":1}
I20260811 11:51:10.180711 19377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874: Neither blocks nor log segments found. Creating new log.
I20260811 11:51:10.181555 19377 log.cc:824] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874: Log is configured to *not* fsync() on all Append() calls
I20260811 11:51:10.183573 19377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874: No bootstrap required, opened a new log
I20260811 11:51:10.187203 19377 raft_consensus.cc:348] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76ae57301c64ea2856e474d1df8b874" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42889 } }
I20260811 11:51:10.187461 19377 raft_consensus.cc:374] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:51:10.187536 19377 raft_consensus.cc:729] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c76ae57301c64ea2856e474d1df8b874, State: Initialized, Role: FOLLOWER
I20260811 11:51:10.187990 19377 consensus_queue.cc:260] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76ae57301c64ea2856e474d1df8b874" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42889 } }
I20260811 11:51:10.188133 19377 raft_consensus.cc:388] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:51:10.188205 19377 raft_consensus.cc:482] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:51:10.188308 19377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:51:10.189338 19377 raft_consensus.cc:504] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76ae57301c64ea2856e474d1df8b874" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42889 } }
I20260811 11:51:10.189692 19377 leader_election.cc:302] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c76ae57301c64ea2856e474d1df8b874; no voters:
I20260811 11:51:10.190016 19377 leader_election.cc:288] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:51:10.190157 19387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:51:10.190411 19387 raft_consensus.cc:686] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874 [term 1 LEADER]: Becoming Leader. State: Replica: c76ae57301c64ea2856e474d1df8b874, State: Running, Role: LEADER
I20260811 11:51:10.190768 19387 consensus_queue.cc:237] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76ae57301c64ea2856e474d1df8b874" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42889 } }
I20260811 11:51:10.191315 19377 sys_catalog.cc:564] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:51:10.192934 19390 sys_catalog.cc:455] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c76ae57301c64ea2856e474d1df8b874. Latest consensus state: current_term: 1 leader_uuid: "c76ae57301c64ea2856e474d1df8b874" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76ae57301c64ea2856e474d1df8b874" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42889 } } }
I20260811 11:51:10.193048 19390 sys_catalog.cc:458] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874 [sys.catalog]: This master's current role is: LEADER
I20260811 11:51:10.193333 19389 sys_catalog.cc:455] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c76ae57301c64ea2856e474d1df8b874" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76ae57301c64ea2856e474d1df8b874" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42889 } } }
I20260811 11:51:10.193436 19389 sys_catalog.cc:458] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874 [sys.catalog]: This master's current role is: LEADER
I20260811 11:51:10.193486 19404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:51:10.196942 19404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:51:10.203177 19404 catalog_manager.cc:1347] Generated new cluster ID: f88b0d06b6614d998acd95b065fc6cef
I20260811 11:51:10.203261 19404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:51:10.229159 19404 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:51:10.230525 19404 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:51:10.240139 19404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c76ae57301c64ea2856e474d1df8b874: Generated new TSK 0
I20260811 11:51:10.241091 19404 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:51:10.247377 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:42889
--builtin_ntp_servers=127.3.221.84:34731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260811 11:51:10.369755 19422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:51:10.370071 19422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:51:10.370116 19422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:51:10.374398 19422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:51:10.374473 19422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:51:10.374531 19422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:51:10.379537 19422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:42889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:51:10.380780 19422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:51:10.382349 19422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:51:10.393076 19422 server_base.cc:1034] running on GCE node
W20260811 11:51:10.393049 19435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:51:10.393030 19430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:51:10.393031 19432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:51:10.393956 19422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:51:10.394745 19422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:51:10.395941 19422 hybrid_clock.cc:648] HybridClock initialized: now 1786449070395920 us; error 35 us; skew 500 ppm
I20260811 11:51:10.398083 19422 webserver.cc:460] Webserver started at http://127.3.221.65:42771/ using document root <none> and password file <none>
I20260811 11:51:10.398702 19422 fs_manager.cc:359] Metadata directory not provided
I20260811 11:51:10.398779 19422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:51:10.399014 19422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:51:10.401715 19422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "62ba5e0155c645bb9691c1aebb404605"
format_stamp: "Formatted at 2026-08-11 11:51:10 on dist-test-slave-bndk"
server_key: "5e1f6b535f75690b20e58194cf52f65e"
server_key_iv: "f1fd1672ea1d2b40b4ba1b4bd7e7ddb5"
server_key_version: "encryptionkey@0"
I20260811 11:51:10.402336 19422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "62ba5e0155c645bb9691c1aebb404605"
format_stamp: "Formatted at 2026-08-11 11:51:10 on dist-test-slave-bndk"
server_key: "5e1f6b535f75690b20e58194cf52f65e"
server_key_iv: "f1fd1672ea1d2b40b4ba1b4bd7e7ddb5"
server_key_version: "encryptionkey@0"
I20260811 11:51:10.406468 19422 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 11:51:10.409476 19444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:10.411080 19422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:51:10.411233 19422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "62ba5e0155c645bb9691c1aebb404605"
format_stamp: "Formatted at 2026-08-11 11:51:10 on dist-test-slave-bndk"
server_key: "5e1f6b535f75690b20e58194cf52f65e"
server_key_iv: "f1fd1672ea1d2b40b4ba1b4bd7e7ddb5"
server_key_version: "encryptionkey@0"
I20260811 11:51:10.411362 19422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:51:10.450897 19422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:51:10.451712 19422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:51:10.451916 19422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:51:10.452569 19422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:51:10.453639 19422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:51:10.453691 19422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:10.453738 19422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:51:10.453768 19422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:10.462431 19422 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:42807
I20260811 11:51:10.462463 19619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:42807 every 8 connection(s)
I20260811 11:51:10.463414 19422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:51:10.465859 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 19422
I20260811 11:51:10.466051 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:51:10.466490 3957 external_mini_cluster.cc:1428] Setting key 74354179755f43210acfabbee578dc74
I20260811 11:51:10.469205 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:42889
--builtin_ntp_servers=127.3.221.84:34731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260811 11:51:10.474524 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:10.465481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:44313 (local address 127.3.221.126:42889)
0811 11:51:10.465669 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:10.465674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:10.467134 (+ 1460us) server_negotiation.cc:407] Connection header received
0811 11:51:10.468784 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:10.468787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:10.468855 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:10.468957 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:10.470287 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:10.471019 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:10.472258 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:10.472492 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:10.472584 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:10.473876 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:51:10.473880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:51:10.473882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:51:10.473896 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:51:10.473919 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:51:10.473927 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:51:10.473995 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 11:51:10.474286 (+ 291us) server_negotiation.cc:299] Negotiation successful
0811 11:51:10.474374 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 11:51:10.475740 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:10.465949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:42889 (local address 127.3.221.65:44313)
0811 11:51:10.466947 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:10.467008 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:10.468507 (+ 1499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:10.469201 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:10.469212 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:10.469375 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:10.470119 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:10.470141 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:10.471163 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:10.471167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:10.472123 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:10.472134 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:10.472650 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:10.472654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:10.472828 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:10.473539 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:51:10.473541 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:10.473591 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:51:10.473673 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:51:10.474036 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:51:10.474049 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:51:10.474056 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 11:51:10.474208 (+ 152us) client_negotiation.cc:240] Negotiation successful
0811 11:51:10.474574 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":164,"threads_started":1}
I20260811 11:51:10.477365 19620 heartbeater.cc:347] Connected to a master server at 127.3.221.126:42889
I20260811 11:51:10.477633 19620 heartbeater.cc:464] Registering TS with master...
I20260811 11:51:10.478232 19620 heartbeater.cc:511] Master 127.3.221.126:42889 requested a full tablet report, sending...
I20260811 11:51:10.480079 19324 ts_manager.cc:194] Registered new tserver with Master: 62ba5e0155c645bb9691c1aebb404605 (127.3.221.65:42807)
I20260811 11:51:10.481819 19324 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.221.65:44313
W20260811 11:51:10.590678 19625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:51:10.590998 19625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:51:10.591044 19625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:51:10.595357 19625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:51:10.595430 19625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:51:10.595489 19625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:51:10.600466 19625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:42889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:51:10.601661 19625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:51:10.603089 19625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:51:10.613122 19641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:51:10.613309 19625 server_base.cc:1034] running on GCE node
W20260811 11:51:10.613116 19636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:51:10.613116 19634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:51:10.613826 19625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:51:10.614482 19625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:51:10.615690 19625 hybrid_clock.cc:648] HybridClock initialized: now 1786449070615663 us; error 40 us; skew 500 ppm
I20260811 11:51:10.617648 19625 webserver.cc:460] Webserver started at http://127.3.221.66:39007/ using document root <none> and password file <none>
I20260811 11:51:10.618242 19625 fs_manager.cc:359] Metadata directory not provided
I20260811 11:51:10.618340 19625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:51:10.618556 19625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:51:10.620709 19625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "29bf6854317340979af0ea74f651c070"
format_stamp: "Formatted at 2026-08-11 11:51:10 on dist-test-slave-bndk"
server_key: "f2351648da40bd6c652ec65acc39c916"
server_key_iv: "a018eed68a3c894e3323a026306d663e"
server_key_version: "encryptionkey@0"
I20260811 11:51:10.621352 19625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "29bf6854317340979af0ea74f651c070"
format_stamp: "Formatted at 2026-08-11 11:51:10 on dist-test-slave-bndk"
server_key: "f2351648da40bd6c652ec65acc39c916"
server_key_iv: "a018eed68a3c894e3323a026306d663e"
server_key_version: "encryptionkey@0"
I20260811 11:51:10.625319 19625 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 11:51:10.628104 19652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:10.629477 19625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:51:10.629601 19625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "29bf6854317340979af0ea74f651c070"
format_stamp: "Formatted at 2026-08-11 11:51:10 on dist-test-slave-bndk"
server_key: "f2351648da40bd6c652ec65acc39c916"
server_key_iv: "a018eed68a3c894e3323a026306d663e"
server_key_version: "encryptionkey@0"
I20260811 11:51:10.629715 19625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:51:10.642831 19625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:51:10.643591 19625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:51:10.643786 19625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:51:10.644436 19625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:51:10.645570 19625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:51:10.645629 19625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:10.645685 19625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:51:10.645709 19625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:10.654989 19625 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:36919
I20260811 11:51:10.655049 19828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:36919 every 8 connection(s)
I20260811 11:51:10.656023 19625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:51:10.657478 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 19625
I20260811 11:51:10.657588 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:51:10.657945 3957 external_mini_cluster.cc:1428] Setting key d81f3c62f06a97464f04ec70e613e33c
I20260811 11:51:10.660176 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:42889
--builtin_ntp_servers=127.3.221.84:34731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260811 11:51:10.666349 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:10.658082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:57235 (local address 127.3.221.126:42889)
0811 11:51:10.658210 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:10.658214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:10.659688 (+ 1474us) server_negotiation.cc:407] Connection header received
0811 11:51:10.661355 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:10.661359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:10.661415 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:10.661492 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:10.662674 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:10.663202 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:10.664456 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:10.664679 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:10.664782 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:10.665873 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:51:10.665876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:51:10.665878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:51:10.665890 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:51:10.665910 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:51:10.665917 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:51:10.665977 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 11:51:10.666127 (+ 150us) server_negotiation.cc:299] Negotiation successful
0811 11:51:10.666198 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 11:51:10.667232 19834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:10.658617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:42889 (local address 127.3.221.66:57235)
0811 11:51:10.659532 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:10.659597 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:10.661120 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:10.661672 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:10.661679 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:10.661794 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:10.662509 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:10.662529 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:10.663373 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:10.663376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:10.664325 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:10.664336 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:10.664837 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:10.664841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:10.665004 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:10.665602 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:51:10.665603 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:10.665644 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:51:10.665716 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:51:10.665987 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:51:10.665996 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:51:10.666003 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 11:51:10.666117 (+ 114us) client_negotiation.cc:240] Negotiation successful
0811 11:51:10.666418 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":148,"threads_started":1}
I20260811 11:51:10.668596 19830 heartbeater.cc:347] Connected to a master server at 127.3.221.126:42889
I20260811 11:51:10.668833 19830 heartbeater.cc:464] Registering TS with master...
I20260811 11:51:10.669418 19830 heartbeater.cc:511] Master 127.3.221.126:42889 requested a full tablet report, sending...
I20260811 11:51:10.670598 19324 ts_manager.cc:194] Registered new tserver with Master: 29bf6854317340979af0ea74f651c070 (127.3.221.66:36919)
I20260811 11:51:10.671401 19324 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.221.66:57235
W20260811 11:51:10.779649 19837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:51:10.779942 19837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:51:10.779981 19837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:51:10.784094 19837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:51:10.784168 19837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:51:10.784224 19837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:51:10.789065 19837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:42889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:51:10.790225 19837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:51:10.791678 19837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:51:10.801795 19850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:51:10.801810 19844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:51:10.801877 19837 server_base.cc:1034] running on GCE node
W20260811 11:51:10.801823 19847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:51:10.802495 19837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:51:10.803181 19837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:51:10.804337 19837 hybrid_clock.cc:648] HybridClock initialized: now 1786449070804308 us; error 43 us; skew 500 ppm
I20260811 11:51:10.806211 19837 webserver.cc:460] Webserver started at http://127.3.221.67:39769/ using document root <none> and password file <none>
I20260811 11:51:10.806813 19837 fs_manager.cc:359] Metadata directory not provided
I20260811 11:51:10.806885 19837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:51:10.807101 19837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:51:10.809196 19837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "36649fe02438426aabeaee1075af4095"
format_stamp: "Formatted at 2026-08-11 11:51:10 on dist-test-slave-bndk"
server_key: "56a112061083af3cdec94d1236f2904a"
server_key_iv: "ac6b653e3e085b1ce49551e427b17d59"
server_key_version: "encryptionkey@0"
I20260811 11:51:10.809803 19837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "36649fe02438426aabeaee1075af4095"
format_stamp: "Formatted at 2026-08-11 11:51:10 on dist-test-slave-bndk"
server_key: "56a112061083af3cdec94d1236f2904a"
server_key_iv: "ac6b653e3e085b1ce49551e427b17d59"
server_key_version: "encryptionkey@0"
I20260811 11:51:10.813499 19837 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260811 11:51:10.816125 19862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:10.817406 19837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 11:51:10.817539 19837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "36649fe02438426aabeaee1075af4095"
format_stamp: "Formatted at 2026-08-11 11:51:10 on dist-test-slave-bndk"
server_key: "56a112061083af3cdec94d1236f2904a"
server_key_iv: "ac6b653e3e085b1ce49551e427b17d59"
server_key_version: "encryptionkey@0"
I20260811 11:51:10.817641 19837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:51:10.837162 19837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:51:10.837966 19837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:51:10.838141 19837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:51:10.838738 19837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:51:10.839700 19837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:51:10.839748 19837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:10.839803 19837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:51:10.839828 19837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:10.848680 19837 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:46187
I20260811 11:51:10.848704 20027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:46187 every 8 connection(s)
I20260811 11:51:10.849668 19837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:51:10.858165 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 19837
I20260811 11:51:10.858279 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:51:10.858547 3957 external_mini_cluster.cc:1428] Setting key 7c8b382c3aa98516f4e367381cd8ba60
I20260811 11:51:10.858808 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:10.851583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:58173 (local address 127.3.221.126:42889)
0811 11:51:10.851717 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:10.851721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:10.852853 (+ 1132us) server_negotiation.cc:407] Connection header received
0811 11:51:10.854232 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:10.854236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:10.854301 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:10.854373 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:10.855300 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:10.855821 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:10.856941 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:10.857154 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:10.857271 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:10.858331 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:51:10.858335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:51:10.858338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:51:10.858352 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:51:10.858376 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:51:10.858385 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:51:10.858444 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 11:51:10.858595 (+ 151us) server_negotiation.cc:299] Negotiation successful
0811 11:51:10.858659 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 11:51:10.859792 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:10.851983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:42889 (local address 127.3.221.67:58173)
0811 11:51:10.852676 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:10.852719 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:10.854021 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:10.854560 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:10.854566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:10.854652 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:10.855151 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:10.855163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:10.855945 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:10.855948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:10.856818 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:10.856828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:10.857322 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:10.857326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:10.857480 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:10.858064 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:51:10.858065 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:10.858106 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:51:10.858172 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:51:10.858458 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:51:10.858468 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:51:10.858474 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 11:51:10.858582 (+ 108us) client_negotiation.cc:240] Negotiation successful
0811 11:51:10.858880 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":131,"threads_started":1}
I20260811 11:51:10.860862 20028 heartbeater.cc:347] Connected to a master server at 127.3.221.126:42889
I20260811 11:51:10.861088 20028 heartbeater.cc:464] Registering TS with master...
I20260811 11:51:10.861652 20028 heartbeater.cc:511] Master 127.3.221.126:42889 requested a full tablet report, sending...
I20260811 11:51:10.862697 19323 ts_manager.cc:194] Registered new tserver with Master: 36649fe02438426aabeaee1075af4095 (127.3.221.67:46187)
I20260811 11:51:10.863273 19323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.221.67:58173
I20260811 11:51:10.872318 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 11:51:10.875293 19385 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:10.874330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33836 (local address 127.3.221.126:42889)
0811 11:51:10.874456 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:10.874460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:10.874653 (+ 193us) server_negotiation.cc:407] Connection header received
0811 11:51:10.874796 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:10.874799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:10.874841 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:10.874903 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:10.875178 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33836: BlockingRecv error: recv got EOF from 127.0.0.1:33836 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 11:51:10.875486 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 19422
I20260811 11:51:10.896315 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 19625
I20260811 11:51:10.914754 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 19837
I20260811 11:51:10.944748 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 19271
2026-08-11T11:51:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1000 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-11T11:51:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:51:10Z Disabled control of system clock
I20260811 11:51:10.980639 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:42545
--webserver_interface=127.3.221.126
--webserver_port=0
--builtin_ntp_servers=127.3.221.84:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:42545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260811 11:51:11.096532 20043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:51:11.096815 20043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:51:11.096855 20043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:51:11.100937 20043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:51:11.101018 20043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:51:11.101033 20043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:51:11.101048 20043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:51:11.101063 20043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:51:11.105984 20043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:42545
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:42545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:51:11.107149 20043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:51:11.108561 20043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:51:11.117089 20059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:51:11.117129 20054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:51:11.117097 20056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:51:11.117260 20043 server_base.cc:1034] running on GCE node
I20260811 11:51:11.117930 20043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:51:11.119055 20043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:51:11.120198 20043 hybrid_clock.cc:648] HybridClock initialized: now 1786449071120175 us; error 43 us; skew 500 ppm
I20260811 11:51:11.122159 20043 webserver.cc:460] Webserver started at http://127.3.221.126:42167/ using document root <none> and password file <none>
I20260811 11:51:11.122746 20043 fs_manager.cc:359] Metadata directory not provided
I20260811 11:51:11.122812 20043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:51:11.123018 20043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:51:11.125295 20043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "6d2fd9e930f8429fb6500669e12daa7c"
format_stamp: "Formatted at 2026-08-11 11:51:11 on dist-test-slave-bndk"
server_key: "3ec7c254296fe827814245d1fa504793"
server_key_iv: "be20675d70344d2863ad9849ece31cdd"
server_key_version: "encryptionkey@0"
I20260811 11:51:11.125890 20043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "6d2fd9e930f8429fb6500669e12daa7c"
format_stamp: "Formatted at 2026-08-11 11:51:11 on dist-test-slave-bndk"
server_key: "3ec7c254296fe827814245d1fa504793"
server_key_iv: "be20675d70344d2863ad9849ece31cdd"
server_key_version: "encryptionkey@0"
I20260811 11:51:11.129946 20043 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 11:51:11.132929 20070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:11.134434 20043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:51:11.134568 20043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "6d2fd9e930f8429fb6500669e12daa7c"
format_stamp: "Formatted at 2026-08-11 11:51:11 on dist-test-slave-bndk"
server_key: "3ec7c254296fe827814245d1fa504793"
server_key_iv: "be20675d70344d2863ad9849ece31cdd"
server_key_version: "encryptionkey@0"
I20260811 11:51:11.134698 20043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:51:11.139590 20043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:51:11.140240 20043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:51:11.140405 20043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:51:11.147503 20043 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:42545
I20260811 11:51:11.147537 20140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:42545 every 8 connection(s)
I20260811 11:51:11.148470 20043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:51:11.151861 20141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:51:11.157027 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 20043
I20260811 11:51:11.157133 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:51:11.157440 3957 external_mini_cluster.cc:1428] Setting key 14ede87e0345c20dab686ffbd07a6db9
I20260811 11:51:11.157831 20141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c: Bootstrap starting.
I20260811 11:51:11.160352 20141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c: Neither blocks nor log segments found. Creating new log.
I20260811 11:51:11.161239 20141 log.cc:824] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c: Log is configured to *not* fsync() on all Append() calls
I20260811 11:51:11.163084 20146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:11.158919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53608 (local address 127.3.221.126:42545)
0811 11:51:11.159554 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:11.159570 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:11.159607 (+ 37us) server_negotiation.cc:407] Connection header received
0811 11:51:11.160272 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:11.160290 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:11.160678 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:11.160980 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:11.161832 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:51:11.161836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:51:11.161850 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:51:11.161861 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:51:11.161920 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:51:11.161939 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:51:11.162039 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 11:51:11.162149 (+ 110us) server_negotiation.cc:299] Negotiation successful
0811 11:51:11.162391 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":118,"threads_started":1}
I20260811 11:51:11.163301 20141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c: No bootstrap required, opened a new log
I20260811 11:51:11.166934 20141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2fd9e930f8429fb6500669e12daa7c" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42545 } }
I20260811 11:51:11.167179 20141 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:51:11.167238 20141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d2fd9e930f8429fb6500669e12daa7c, State: Initialized, Role: FOLLOWER
I20260811 11:51:11.167661 20141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2fd9e930f8429fb6500669e12daa7c" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42545 } }
I20260811 11:51:11.167809 20141 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:51:11.167873 20141 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:51:11.167976 20141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:51:11.168917 20141 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2fd9e930f8429fb6500669e12daa7c" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42545 } }
I20260811 11:51:11.169286 20141 leader_election.cc:302] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d2fd9e930f8429fb6500669e12daa7c; no voters:
I20260811 11:51:11.169615 20141 leader_election.cc:288] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:51:11.169723 20150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:51:11.169976 20150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c [term 1 LEADER]: Becoming Leader. State: Replica: 6d2fd9e930f8429fb6500669e12daa7c, State: Running, Role: LEADER
I20260811 11:51:11.170342 20150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2fd9e930f8429fb6500669e12daa7c" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42545 } }
I20260811 11:51:11.170866 20141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:51:11.172078 20151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d2fd9e930f8429fb6500669e12daa7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2fd9e930f8429fb6500669e12daa7c" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42545 } } }
I20260811 11:51:11.172219 20151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c [sys.catalog]: This master's current role is: LEADER
I20260811 11:51:11.172562 20153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d2fd9e930f8429fb6500669e12daa7c. Latest consensus state: current_term: 1 leader_uuid: "6d2fd9e930f8429fb6500669e12daa7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2fd9e930f8429fb6500669e12daa7c" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42545 } } }
I20260811 11:51:11.172616 20165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:51:11.173545 20153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c [sys.catalog]: This master's current role is: LEADER
I20260811 11:51:11.176028 20165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:51:11.182165 20165 catalog_manager.cc:1347] Generated new cluster ID: a130fbf22c0c46b2a4a186a82bffa368
I20260811 11:51:11.182255 20165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:51:11.189925 20165 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:51:11.190240 20165 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:51:11.199020 20165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d2fd9e930f8429fb6500669e12daa7c: Generated new TSK 0
I20260811 11:51:11.199729 20165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:51:11.205559 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:42545
--builtin_ntp_servers=127.3.221.84:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260811 11:51:11.322717 20181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:51:11.323006 20181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:51:11.323047 20181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:51:11.327306 20181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:51:11.327369 20181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:51:11.327436 20181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:51:11.332468 20181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:42545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:51:11.333679 20181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:51:11.335156 20181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:51:11.345247 20192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:51:11.345377 20181 server_base.cc:1034] running on GCE node
W20260811 11:51:11.345263 20194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:51:11.345304 20189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:51:11.345932 20181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:51:11.346604 20181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:51:11.347788 20181 hybrid_clock.cc:648] HybridClock initialized: now 1786449071347757 us; error 43 us; skew 500 ppm
I20260811 11:51:11.349803 20181 webserver.cc:460] Webserver started at http://127.3.221.65:38181/ using document root <none> and password file <none>
I20260811 11:51:11.350396 20181 fs_manager.cc:359] Metadata directory not provided
I20260811 11:51:11.350471 20181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:51:11.350688 20181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:51:11.352934 20181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "00f9473944cf46ca9fcc723cd2934d04"
format_stamp: "Formatted at 2026-08-11 11:51:11 on dist-test-slave-bndk"
server_key: "038ef3fd7a961329a06e081286678d43"
server_key_iv: "9d41df1ee0b9f41ade560363227f248f"
server_key_version: "encryptionkey@0"
I20260811 11:51:11.353545 20181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "00f9473944cf46ca9fcc723cd2934d04"
format_stamp: "Formatted at 2026-08-11 11:51:11 on dist-test-slave-bndk"
server_key: "038ef3fd7a961329a06e081286678d43"
server_key_iv: "9d41df1ee0b9f41ade560363227f248f"
server_key_version: "encryptionkey@0"
I20260811 11:51:11.357448 20181 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 11:51:11.360172 20204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:11.361580 20181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:51:11.361706 20181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "00f9473944cf46ca9fcc723cd2934d04"
format_stamp: "Formatted at 2026-08-11 11:51:11 on dist-test-slave-bndk"
server_key: "038ef3fd7a961329a06e081286678d43"
server_key_iv: "9d41df1ee0b9f41ade560363227f248f"
server_key_version: "encryptionkey@0"
I20260811 11:51:11.361835 20181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:51:11.366642 20181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:51:11.367303 20181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:51:11.367467 20181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:51:11.368041 20181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:51:11.368983 20181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:51:11.369030 20181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:11.369086 20181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:51:11.369112 20181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:11.378325 20181 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:38337
I20260811 11:51:11.378378 20387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:38337 every 8 connection(s)
I20260811 11:51:11.379310 20181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:51:11.383190 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 20181
I20260811 11:51:11.383296 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:51:11.383592 3957 external_mini_cluster.cc:1428] Setting key 29a4d9d750bc39038a442238ac4da769
I20260811 11:51:11.385798 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:42545
--builtin_ntp_servers=127.3.221.84:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260811 11:51:11.386142 20146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:11.381353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:56203 (local address 127.3.221.126:42545)
0811 11:51:11.381520 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:11.381525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:11.382859 (+ 1334us) server_negotiation.cc:407] Connection header received
0811 11:51:11.384280 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:11.384284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:11.384333 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:11.384424 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:11.385671 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:51:11.385675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:51:11.385679 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:51:11.385695 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:51:11.385724 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:51:11.385736 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:51:11.385851 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 11:51:11.385960 (+ 109us) server_negotiation.cc:299] Negotiation successful
0811 11:51:11.386012 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 11:51:11.387132 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:11.381811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:42545 (local address 127.3.221.65:56203)
0811 11:51:11.382689 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:11.382742 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:11.384062 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:11.384593 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:11.384600 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:11.384701 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:11.385342 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:51:11.385344 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:11.385391 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:51:11.385495 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:51:11.385818 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:51:11.385823 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:51:11.385827 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 11:51:11.386008 (+ 181us) client_negotiation.cc:240] Negotiation successful
0811 11:51:11.386288 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":134,"threads_started":1}
I20260811 11:51:11.388520 20389 heartbeater.cc:347] Connected to a master server at 127.3.221.126:42545
I20260811 11:51:11.388813 20389 heartbeater.cc:464] Registering TS with master...
I20260811 11:51:11.389297 20389 heartbeater.cc:511] Master 127.3.221.126:42545 requested a full tablet report, sending...
I20260811 11:51:11.391245 20090 ts_manager.cc:194] Registered new tserver with Master: 00f9473944cf46ca9fcc723cd2934d04 (127.3.221.65:38337)
W20260811 11:51:11.504712 20394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:51:11.504997 20394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:51:11.505038 20394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:51:11.509068 20394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:51:11.509135 20394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:51:11.509193 20394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:51:11.514263 20394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:42545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:51:11.515360 20394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:51:11.516716 20394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:51:11.526649 20403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:51:11.526649 20405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:51:11.526660 20402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:51:11.526865 20394 server_base.cc:1034] running on GCE node
I20260811 11:51:11.527298 20394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:51:11.527956 20394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:51:11.529119 20394 hybrid_clock.cc:648] HybridClock initialized: now 1786449071529092 us; error 42 us; skew 500 ppm
I20260811 11:51:11.531035 20394 webserver.cc:460] Webserver started at http://127.3.221.66:44651/ using document root <none> and password file <none>
I20260811 11:51:11.531616 20394 fs_manager.cc:359] Metadata directory not provided
I20260811 11:51:11.531678 20394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:51:11.531847 20394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:51:11.534075 20394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "c649244f51f946f39275fae033658768"
format_stamp: "Formatted at 2026-08-11 11:51:11 on dist-test-slave-bndk"
server_key: "ec640f916a1db33c1be242353a59fef1"
server_key_iv: "a0ca46310d6985ec3d5985a415827425"
server_key_version: "encryptionkey@0"
I20260811 11:51:11.534607 20394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "c649244f51f946f39275fae033658768"
format_stamp: "Formatted at 2026-08-11 11:51:11 on dist-test-slave-bndk"
server_key: "ec640f916a1db33c1be242353a59fef1"
server_key_iv: "a0ca46310d6985ec3d5985a415827425"
server_key_version: "encryptionkey@0"
I20260811 11:51:11.538349 20394 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:51:11.541028 20412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:11.542403 20394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:51:11.542526 20394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "c649244f51f946f39275fae033658768"
format_stamp: "Formatted at 2026-08-11 11:51:11 on dist-test-slave-bndk"
server_key: "ec640f916a1db33c1be242353a59fef1"
server_key_iv: "a0ca46310d6985ec3d5985a415827425"
server_key_version: "encryptionkey@0"
I20260811 11:51:11.542645 20394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:51:11.547448 20394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:51:11.548091 20394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:51:11.548254 20394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:51:11.548803 20394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:51:11.549810 20394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:51:11.549858 20394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:11.549899 20394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:51:11.549928 20394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:11.558583 20394 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:43565
I20260811 11:51:11.558609 20597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:43565 every 8 connection(s)
I20260811 11:51:11.559510 20394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:51:11.564031 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 20394
I20260811 11:51:11.564167 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:51:11.564472 3957 external_mini_cluster.cc:1428] Setting key c64e25bb4037991631c8681f1073d4db
I20260811 11:51:11.565968 20146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:11.561341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:49075 (local address 127.3.221.126:42545)
0811 11:51:11.561468 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:11.561472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:11.562616 (+ 1144us) server_negotiation.cc:407] Connection header received
0811 11:51:11.563985 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:11.563989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:11.564034 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:11.564124 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:11.565480 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:51:11.565484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:51:11.565486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:51:11.565501 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:51:11.565523 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:51:11.565532 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:51:11.565612 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 11:51:11.565783 (+ 171us) server_negotiation.cc:299] Negotiation successful
0811 11:51:11.565832 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 11:51:11.566648 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:42545
--builtin_ntp_servers=127.3.221.84:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260811 11:51:11.567075 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:11.561765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:42545 (local address 127.3.221.66:49075)
0811 11:51:11.562461 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:11.562504 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:11.563707 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:11.564337 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:11.564348 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:11.564482 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:11.565157 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:51:11.565158 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:11.565203 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:51:11.565326 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:51:11.565605 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:51:11.565613 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:51:11.565620 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 11:51:11.565795 (+ 175us) client_negotiation.cc:240] Negotiation successful
0811 11:51:11.566048 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":128,"threads_started":1}
I20260811 11:51:11.568202 20598 heartbeater.cc:347] Connected to a master server at 127.3.221.126:42545
I20260811 11:51:11.568483 20598 heartbeater.cc:464] Registering TS with master...
I20260811 11:51:11.568903 20598 heartbeater.cc:511] Master 127.3.221.126:42545 requested a full tablet report, sending...
I20260811 11:51:11.570050 20090 ts_manager.cc:194] Registered new tserver with Master: c649244f51f946f39275fae033658768 (127.3.221.66:43565)
W20260811 11:51:11.687098 20602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:51:11.687379 20602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:51:11.687419 20602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:51:11.692039 20602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:51:11.692116 20602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:51:11.692198 20602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:51:11.698637 20602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:42545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:51:11.699834 20602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:51:11.701292 20602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:51:11.711084 20608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:51:11.710994 20611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:51:11.711232 20602 server_base.cc:1034] running on GCE node
W20260811 11:51:11.711002 20607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:51:11.711798 20602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:51:11.712481 20602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:51:11.713640 20602 hybrid_clock.cc:648] HybridClock initialized: now 1786449071713625 us; error 33 us; skew 500 ppm
I20260811 11:51:11.715525 20602 webserver.cc:460] Webserver started at http://127.3.221.67:34651/ using document root <none> and password file <none>
I20260811 11:51:11.716111 20602 fs_manager.cc:359] Metadata directory not provided
I20260811 11:51:11.716183 20602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:51:11.716398 20602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:51:11.718562 20602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "b5379da87c3b4268887a8c1d7840e15e"
format_stamp: "Formatted at 2026-08-11 11:51:11 on dist-test-slave-bndk"
server_key: "8d35af5a7e7a291326a2c6adf021cab3"
server_key_iv: "23d63edde6e6d476e865443c12e5e2ff"
server_key_version: "encryptionkey@0"
I20260811 11:51:11.719125 20602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "b5379da87c3b4268887a8c1d7840e15e"
format_stamp: "Formatted at 2026-08-11 11:51:11 on dist-test-slave-bndk"
server_key: "8d35af5a7e7a291326a2c6adf021cab3"
server_key_iv: "23d63edde6e6d476e865443c12e5e2ff"
server_key_version: "encryptionkey@0"
I20260811 11:51:11.722997 20602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 11:51:11.725618 20617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:11.726915 20602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 11:51:11.727027 20602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "b5379da87c3b4268887a8c1d7840e15e"
format_stamp: "Formatted at 2026-08-11 11:51:11 on dist-test-slave-bndk"
server_key: "8d35af5a7e7a291326a2c6adf021cab3"
server_key_iv: "23d63edde6e6d476e865443c12e5e2ff"
server_key_version: "encryptionkey@0"
I20260811 11:51:11.727129 20602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:51:11.731817 20602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:51:11.732450 20602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:51:11.732606 20602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:51:11.733274 20602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:51:11.734251 20602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:51:11.734299 20602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:11.734352 20602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:51:11.734380 20602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:11.743021 20602 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:35945
I20260811 11:51:11.743045 20802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:35945 every 8 connection(s)
I20260811 11:51:11.743933 20602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:51:11.744518 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 20602
I20260811 11:51:11.744630 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:51:11.744896 3957 external_mini_cluster.cc:1428] Setting key a71f8570545003390c88ec87da0be099
I20260811 11:51:11.750391 20146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:11.745831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:46935 (local address 127.3.221.126:42545)
0811 11:51:11.745953 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:11.745963 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:11.747182 (+ 1219us) server_negotiation.cc:407] Connection header received
0811 11:51:11.748753 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:11.748757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:11.748803 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:11.748874 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:11.749945 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:51:11.749949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:51:11.749952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:51:11.749965 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:51:11.749989 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:51:11.749997 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:51:11.750072 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 11:51:11.750207 (+ 135us) server_negotiation.cc:299] Negotiation successful
0811 11:51:11.750253 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260811 11:51:11.751289 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:11.746306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:42545 (local address 127.3.221.67:46935)
0811 11:51:11.747041 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:11.747085 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:11.748550 (+ 1465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:11.749055 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:11.749062 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:11.749155 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:11.749682 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:51:11.749684 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:11.749725 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:51:11.749799 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:51:11.750067 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:51:11.750076 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:51:11.750082 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 11:51:11.750247 (+ 165us) client_negotiation.cc:240] Negotiation successful
0811 11:51:11.750499 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":114,"threads_started":1}
I20260811 11:51:11.752534 20803 heartbeater.cc:347] Connected to a master server at 127.3.221.126:42545
I20260811 11:51:11.752767 20803 heartbeater.cc:464] Registering TS with master...
I20260811 11:51:11.753157 20803 heartbeater.cc:511] Master 127.3.221.126:42545 requested a full tablet report, sending...
I20260811 11:51:11.754264 20090 ts_manager.cc:194] Registered new tserver with Master: b5379da87c3b4268887a8c1d7840e15e (127.3.221.67:35945)
I20260811 11:51:11.760221 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 11:51:11.763053 20146 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:51:11.762174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53620 (local address 127.3.221.126:42545)
0811 11:51:11.762313 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:11.762347 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:11.762489 (+ 142us) server_negotiation.cc:407] Connection header received
0811 11:51:11.762604 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:11.762606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:11.762641 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:11.762703 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:11.762920 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53620: BlockingRecv error: recv got EOF from 127.0.0.1:53620 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 11:51:11.763211 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 20181
I20260811 11:51:11.775782 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 20394
I20260811 11:51:11.785876 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 20602
I20260811 11:51:11.804733 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 20043
2026-08-11T11:51:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (864 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:51:11 dist-test-slave-bndk krb5kdc[20817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:51:11 dist-test-slave-bndk krb5kdc[20817](info): set up 2 sockets
Aug 11 11:51:11 dist-test-slave-bndk krb5kdc[20817](info): commencing operation
krb5kdc: starting...
W20260811 11:51:12.978116 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:51:13 dist-test-slave-bndk krb5kdc[20817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:51:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:51:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:51:13.112093 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:33093
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:33093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:51:13.227670 20834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:51:13.227991 20834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:51:13.228052 20834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:51:13.232519 20834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:51:13.232594 20834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:51:13.232620 20834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:51:13.232642 20834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:51:13.232671 20834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:51:13.238349 20834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:33093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:33093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:51:13.239511 20834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:51:13.240573 20834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:51:13.249390 20839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:51:13.249447 20834 server_base.cc:1034] running on GCE node
W20260811 11:51:13.249390 20844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:51:13.249693 20840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:51:13.250319 20834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:51:13.251466 20834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:51:13.252626 20834 hybrid_clock.cc:648] HybridClock initialized: now 1786449073252607 us; error 34 us; skew 500 ppm
Aug 11 11:51:13 dist-test-slave-bndk krb5kdc[20817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449073, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:51:13.255511 20834 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:51:13.256491 20834 webserver.cc:460] Webserver started at http://127.3.221.126:37989/ using document root <none> and password file <none>
I20260811 11:51:13.257068 20834 fs_manager.cc:359] Metadata directory not provided
I20260811 11:51:13.257139 20834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:51:13.257391 20834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:51:13.259613 20834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "a6d1c0b94e574afb838e14b324e211a6"
format_stamp: "Formatted at 2026-08-11 11:51:13 on dist-test-slave-bndk"
server_key: "515aaf6da4d328cd4eba7f5f3a9bebe7"
server_key_iv: "75e71456f890f4e189059d035356832c"
server_key_version: "encryptionkey@0"
I20260811 11:51:13.260386 20834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "a6d1c0b94e574afb838e14b324e211a6"
format_stamp: "Formatted at 2026-08-11 11:51:13 on dist-test-slave-bndk"
server_key: "515aaf6da4d328cd4eba7f5f3a9bebe7"
server_key_iv: "75e71456f890f4e189059d035356832c"
server_key_version: "encryptionkey@0"
I20260811 11:51:13.264576 20834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 11:51:13.267227 20853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:13.268544 20834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 11:51:13.268680 20834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "a6d1c0b94e574afb838e14b324e211a6"
format_stamp: "Formatted at 2026-08-11 11:51:13 on dist-test-slave-bndk"
server_key: "515aaf6da4d328cd4eba7f5f3a9bebe7"
server_key_iv: "75e71456f890f4e189059d035356832c"
server_key_version: "encryptionkey@0"
I20260811 11:51:13.268786 20834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:51:13.285969 20834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:51:13.289327 20834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:51:13.289541 20834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:51:13.296947 20834 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:33093
I20260811 11:51:13.297250 20938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:33093 every 8 connection(s)
I20260811 11:51:13.298070 20834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:51:13.299863 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 20834
I20260811 11:51:13.299993 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:51:13.300295 3957 external_mini_cluster.cc:1428] Setting key 7b7085478ef902e76490557510b1c1cd
I20260811 11:51:13.301478 20939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:51:13.308355 20939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6: Bootstrap starting.
Aug 11 11:51:13 dist-test-slave-bndk krb5kdc[20817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:51:13.311296 20939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6: Neither blocks nor log segments found. Creating new log.
I20260811 11:51:13.312098 20939 log.cc:824] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6: Log is configured to *not* fsync() on all Append() calls
I20260811 11:51:13.314720 20939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6: No bootstrap required, opened a new log
I20260811 11:51:13.315670 20942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:13.301802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46892 (local address 127.3.221.126:33093)
0811 11:51:13.302242 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:13.302259 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:13.302299 (+ 40us) server_negotiation.cc:407] Connection header received
0811 11:51:13.302998 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:13.303017 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:13.303370 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:13.303671 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:13.304592 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:13.305680 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:13.306842 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:13.307339 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:13.307458 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:13.310382 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:51:13.310388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:51:13.310403 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:51:13.310436 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:51:13.312210 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:51:13.312744 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:51:13.312751 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:51:13.312758 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:51:13.312907 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:51:13.313203 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:51:13.313206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:51:13.313208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:51:13.313827 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:51:13.314024 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0811 11:51:13.314327 (+ 303us) server_negotiation.cc:299] Negotiation successful
0811 11:51:13.314658 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":127,"threads_started":1}
I20260811 11:51:13.318986 20939 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d1c0b94e574afb838e14b324e211a6" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33093 } }
I20260811 11:51:13.319238 20939 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:51:13.319278 20939 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6d1c0b94e574afb838e14b324e211a6, State: Initialized, Role: FOLLOWER
I20260811 11:51:13.319706 20939 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d1c0b94e574afb838e14b324e211a6" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33093 } }
I20260811 11:51:13.319850 20939 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:51:13.319898 20939 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:51:13.319972 20939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:51:13.320925 20939 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d1c0b94e574afb838e14b324e211a6" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33093 } }
I20260811 11:51:13.321333 20939 leader_election.cc:302] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6d1c0b94e574afb838e14b324e211a6; no voters:
I20260811 11:51:13.321646 20939 leader_election.cc:288] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:51:13.321799 20944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:51:13.322147 20944 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6 [term 1 LEADER]: Becoming Leader. State: Replica: a6d1c0b94e574afb838e14b324e211a6, State: Running, Role: LEADER
I20260811 11:51:13.322541 20944 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d1c0b94e574afb838e14b324e211a6" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33093 } }
I20260811 11:51:13.323000 20939 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:51:13.324237 20948 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6d1c0b94e574afb838e14b324e211a6. Latest consensus state: current_term: 1 leader_uuid: "a6d1c0b94e574afb838e14b324e211a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d1c0b94e574afb838e14b324e211a6" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33093 } } }
I20260811 11:51:13.324373 20948 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6 [sys.catalog]: This master's current role is: LEADER
I20260811 11:51:13.324648 20947 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6d1c0b94e574afb838e14b324e211a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d1c0b94e574afb838e14b324e211a6" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33093 } } }
I20260811 11:51:13.324752 20947 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6 [sys.catalog]: This master's current role is: LEADER
I20260811 11:51:13.324853 20956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:51:13.327982 20956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:51:13.334360 20956 catalog_manager.cc:1347] Generated new cluster ID: 2206c0f9332140e2b66ea9e54429d986
I20260811 11:51:13.334446 20956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:51:13.347893 20956 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:51:13.348881 20956 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:51:13.354388 20956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6: Generated new TSK 0
I20260811 11:51:13.355391 20956 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
I20260811 11:51:13.400269 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:33093
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:51:13.536286 20974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:51:13.536578 20974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:51:13.536623 20974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:51:13.540668 20974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:51:13.540738 20974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:51:13.540797 20974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:51:13.546198 20974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:33093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:51:13.547405 20974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:51:13.548576 20974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:51:13.558277 20981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:51:13.558372 20974 server_base.cc:1034] running on GCE node
W20260811 11:51:13.558277 20986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:51:13.558327 20982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:51:13.559005 20974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:51:13.559689 20974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:51:13.560868 20974 hybrid_clock.cc:648] HybridClock initialized: now 1786449073560849 us; error 32 us; skew 500 ppm
Aug 11 11:51:13 dist-test-slave-bndk krb5kdc[20817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449073, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:51:13.563709 20974 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:51:13.564633 20974 webserver.cc:460] Webserver started at http://127.3.221.65:44615/ using document root <none> and password file <none>
I20260811 11:51:13.565201 20974 fs_manager.cc:359] Metadata directory not provided
I20260811 11:51:13.565318 20974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:51:13.565541 20974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:51:13.567633 20974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "e6fc6e7f76c7454caf2912ab46c11bb4"
format_stamp: "Formatted at 2026-08-11 11:51:13 on dist-test-slave-bndk"
server_key: "ec740b72e8bc2a411f1ad3a00aa602d5"
server_key_iv: "aff2674d621605affb67df7fac62ed81"
server_key_version: "encryptionkey@0"
I20260811 11:51:13.568208 20974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "e6fc6e7f76c7454caf2912ab46c11bb4"
format_stamp: "Formatted at 2026-08-11 11:51:13 on dist-test-slave-bndk"
server_key: "ec740b72e8bc2a411f1ad3a00aa602d5"
server_key_iv: "aff2674d621605affb67df7fac62ed81"
server_key_version: "encryptionkey@0"
I20260811 11:51:13.572252 20974 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260811 11:51:13.574987 20998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:13.576285 20974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 11:51:13.576417 20974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "e6fc6e7f76c7454caf2912ab46c11bb4"
format_stamp: "Formatted at 2026-08-11 11:51:13 on dist-test-slave-bndk"
server_key: "ec740b72e8bc2a411f1ad3a00aa602d5"
server_key_iv: "aff2674d621605affb67df7fac62ed81"
server_key_version: "encryptionkey@0"
I20260811 11:51:13.576525 20974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:51:13.600834 20974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:51:13.604219 20974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:51:13.604446 20974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:51:13.605105 20974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:51:13.606038 20974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:51:13.606101 20974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:13.606163 20974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:51:13.606189 20974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:13.614897 20974 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:35291
I20260811 11:51:13.614924 21177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:35291 every 8 connection(s)
I20260811 11:51:13.615861 20974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:51:13.619840 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 20974
I20260811 11:51:13.619997 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:51:13.620304 3957 external_mini_cluster.cc:1428] Setting key c65e2158c296006b3530f98a208c28ff
Aug 11 11:51:13 dist-test-slave-bndk krb5kdc[20817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449073, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.66@KRBTEST.COM" created.
I20260811 11:51:13.633425 20942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:13.617788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:33337 (local address 127.3.221.126:33093)
0811 11:51:13.617950 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:13.617954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:13.618907 (+ 953us) server_negotiation.cc:407] Connection header received
0811 11:51:13.620581 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:13.620585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:13.620639 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:13.620728 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:13.622111 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:13.622691 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:13.623952 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:13.624187 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:13.624280 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:13.629898 (+ 5618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:51:13.629902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:51:13.629904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:51:13.629933 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:51:13.631230 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:51:13.631791 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:51:13.631794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:51:13.631796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:51:13.631888 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:51:13.632231 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:51:13.632234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:51:13.632236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:51:13.632437 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:51:13.632527 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 11:51:13.633073 (+ 546us) server_negotiation.cc:299] Negotiation successful
0811 11:51:13.633250 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 11:51:13.634290 21183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:13.618187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:33093 (local address 127.3.221.65:33337)
0811 11:51:13.618738 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:13.618787 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:13.620313 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:13.620948 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:13.620955 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:13.621431 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:13.621956 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:13.621971 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:13.622830 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:13.622833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:13.623820 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:13.623833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:13.624349 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:13.624353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:13.624523 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:13.627903 (+ 3380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:13.627925 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:13.629660 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:51:13.631382 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:51:13.631390 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:51:13.631406 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:51:13.631681 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:51:13.631977 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:51:13.631980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:51:13.631983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:51:13.632135 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:51:13.632545 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:51:13.632552 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:51:13.632825 (+ 273us) client_negotiation.cc:769] Sending connection context
0811 11:51:13.633009 (+ 184us) client_negotiation.cc:240] Negotiation successful
0811 11:51:13.633406 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":128,"threads_started":1}
I20260811 11:51:13.635814 21178 heartbeater.cc:347] Connected to a master server at 127.3.221.126:33093
I20260811 11:51:13.636076 21178 heartbeater.cc:464] Registering TS with master...
I20260811 11:51:13.636708 21178 heartbeater.cc:511] Master 127.3.221.126:33093 requested a full tablet report, sending...
I20260811 11:51:13.638432 20883 ts_manager.cc:194] Registered new tserver with Master: e6fc6e7f76c7454caf2912ab46c11bb4 (127.3.221.65:35291)
I20260811 11:51:13.639576 20883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:33337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
I20260811 11:51:13.651747 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:33093
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:51:13.770534 21186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:51:13.770810 21186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:51:13.770856 21186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:51:13.775097 21186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:51:13.775168 21186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:51:13.775235 21186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:51:13.781308 21186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:33093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:51:13.782462 21186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:51:13.783538 21186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:51:13.794206 21193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:51:13.794210 21198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:51:13.794265 21186 server_base.cc:1034] running on GCE node
W20260811 11:51:13.794206 21191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:51:13.794839 21186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:51:13.795497 21186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:51:13.796689 21186 hybrid_clock.cc:648] HybridClock initialized: now 1786449073796656 us; error 44 us; skew 500 ppm
Aug 11 11:51:13 dist-test-slave-bndk krb5kdc[20817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449073, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:51:13.799530 21186 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:51:13.800444 21186 webserver.cc:460] Webserver started at http://127.3.221.66:33913/ using document root <none> and password file <none>
I20260811 11:51:13.801023 21186 fs_manager.cc:359] Metadata directory not provided
I20260811 11:51:13.801092 21186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:51:13.801344 21186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:51:13.803486 21186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "91e49a8e21874a26953d0768d1eb84d1"
format_stamp: "Formatted at 2026-08-11 11:51:13 on dist-test-slave-bndk"
server_key: "d3a0a19e56e5a787ea374348cc166b38"
server_key_iv: "24a42d02a04604c0f9431d28e37d2de9"
server_key_version: "encryptionkey@0"
I20260811 11:51:13.804059 21186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "91e49a8e21874a26953d0768d1eb84d1"
format_stamp: "Formatted at 2026-08-11 11:51:13 on dist-test-slave-bndk"
server_key: "d3a0a19e56e5a787ea374348cc166b38"
server_key_iv: "24a42d02a04604c0f9431d28e37d2de9"
server_key_version: "encryptionkey@0"
I20260811 11:51:13.807871 21186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 11:51:13.810441 21211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:13.811739 21186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 11:51:13.811856 21186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "91e49a8e21874a26953d0768d1eb84d1"
format_stamp: "Formatted at 2026-08-11 11:51:13 on dist-test-slave-bndk"
server_key: "d3a0a19e56e5a787ea374348cc166b38"
server_key_iv: "24a42d02a04604c0f9431d28e37d2de9"
server_key_version: "encryptionkey@0"
I20260811 11:51:13.811967 21186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:51:13.828533 21186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:51:13.831454 21186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:51:13.831632 21186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:51:13.832216 21186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:51:13.833065 21186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:51:13.833113 21186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:13.833168 21186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:51:13.833194 21186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:13.841877 21186 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:36483
I20260811 11:51:13.841904 21395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:36483 every 8 connection(s)
I20260811 11:51:13.842792 21186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:51:13.849628 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 21186
I20260811 11:51:13.849745 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:51:13.850011 3957 external_mini_cluster.cc:1428] Setting key f98a8bb47ccf8dadc01d6962e63c4112
Aug 11 11:51:13 dist-test-slave-bndk krb5kdc[20817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449073, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:51:13.857321 20942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:13.844683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:40007 (local address 127.3.221.126:33093)
0811 11:51:13.844858 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:13.844863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:13.845759 (+ 896us) server_negotiation.cc:407] Connection header received
0811 11:51:13.847195 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:13.847198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:13.847248 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:13.847323 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:13.848494 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:13.849058 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:13.850322 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:13.850568 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:13.850654 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:13.853388 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:51:13.853392 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:51:13.853395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:51:13.853427 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:51:13.854840 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:51:13.855448 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:51:13.855452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:51:13.855454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:51:13.855551 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:51:13.855952 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:51:13.855956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:51:13.855957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:51:13.856224 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:51:13.856343 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0811 11:51:13.856949 (+ 606us) server_negotiation.cc:299] Negotiation successful
0811 11:51:13.857097 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 11:51:13.858150 21400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:13.845099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:33093 (local address 127.3.221.66:40007)
0811 11:51:13.845606 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:13.845651 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:13.846990 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:13.847482 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:13.847489 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:13.847850 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:13.848346 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:13.848361 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:13.849182 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:13.849185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:13.850194 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:13.850205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:13.850717 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:13.850722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:13.850879 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:13.851515 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:13.851536 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:13.853166 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:51:13.854977 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:51:13.854983 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:51:13.855001 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:51:13.855333 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:51:13.855638 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:51:13.855641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:51:13.855642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:51:13.855802 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:51:13.856378 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:51:13.856386 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:51:13.856667 (+ 281us) client_negotiation.cc:769] Sending connection context
0811 11:51:13.856851 (+ 184us) client_negotiation.cc:240] Negotiation successful
0811 11:51:13.857186 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":158,"threads_started":1}
I20260811 11:51:13.859356 21396 heartbeater.cc:347] Connected to a master server at 127.3.221.126:33093
I20260811 11:51:13.859596 21396 heartbeater.cc:464] Registering TS with master...
I20260811 11:51:13.860106 21396 heartbeater.cc:511] Master 127.3.221.126:33093 requested a full tablet report, sending...
I20260811 11:51:13.861173 20883 ts_manager.cc:194] Registered new tserver with Master: 91e49a8e21874a26953d0768d1eb84d1 (127.3.221.66:36483)
WARNING: no policy specified for kudu/127.3.221.67@KRBTEST.COM; defaulting to no policy
I20260811 11:51:13.861778 20883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:40007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
I20260811 11:51:13.880962 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:33093
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:51:13.996016 21403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:51:13.996311 21403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:51:13.996373 21403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:51:14.001334 21403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:51:14.001415 21403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:51:14.001503 21403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:51:14.007056 21403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:33093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:51:14.008203 21403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:51:14.009313 21403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:51:14.019183 21408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:51:14.019178 21412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:51:14.019284 21403 server_base.cc:1034] running on GCE node
W20260811 11:51:14.019189 21410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:51:14.019922 21403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:51:14.020578 21403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:51:14.021733 21403 hybrid_clock.cc:648] HybridClock initialized: now 1786449074021702 us; error 42 us; skew 500 ppm
Aug 11 11:51:14 dist-test-slave-bndk krb5kdc[20817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449074, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:51:14.024609 21403 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:51:14.025547 21403 webserver.cc:460] Webserver started at http://127.3.221.67:43507/ using document root <none> and password file <none>
I20260811 11:51:14.026103 21403 fs_manager.cc:359] Metadata directory not provided
I20260811 11:51:14.026170 21403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:51:14.026376 21403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:51:14.028427 21403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "697c879c27354deba3f695e21716ab2d"
format_stamp: "Formatted at 2026-08-11 11:51:14 on dist-test-slave-bndk"
server_key: "3504c1ed3dddd272d550229c9b11799d"
server_key_iv: "a78ab6d5fb0f06dcd66a466adebb5a5a"
server_key_version: "encryptionkey@0"
I20260811 11:51:14.029023 21403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "697c879c27354deba3f695e21716ab2d"
format_stamp: "Formatted at 2026-08-11 11:51:14 on dist-test-slave-bndk"
server_key: "3504c1ed3dddd272d550229c9b11799d"
server_key_iv: "a78ab6d5fb0f06dcd66a466adebb5a5a"
server_key_version: "encryptionkey@0"
I20260811 11:51:14.033011 21403 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260811 11:51:14.035567 21422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:14.036828 21403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 11:51:14.036943 21403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "697c879c27354deba3f695e21716ab2d"
format_stamp: "Formatted at 2026-08-11 11:51:14 on dist-test-slave-bndk"
server_key: "3504c1ed3dddd272d550229c9b11799d"
server_key_iv: "a78ab6d5fb0f06dcd66a466adebb5a5a"
server_key_version: "encryptionkey@0"
I20260811 11:51:14.037111 21403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:51:14.050253 21403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:51:14.053357 21403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:51:14.053545 21403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:51:14.054154 21403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:51:14.055030 21403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:51:14.055080 21403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:14.055135 21403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:51:14.055164 21403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:51:14.064096 21403 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:42547
I20260811 11:51:14.064134 21583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:42547 every 8 connection(s)
I20260811 11:51:14.065066 21403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:51:14.068413 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 21403
I20260811 11:51:14.068558 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:51:14.068897 3957 external_mini_cluster.cc:1428] Setting key 1f2eebc717f7f858ff7a08b6b13b53b7
Aug 11 11:51:14 dist-test-slave-bndk krb5kdc[20817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449074, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:51:14.080055 20942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:14.067085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:47195 (local address 127.3.221.126:33093)
0811 11:51:14.067271 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:14.067276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:14.068215 (+ 939us) server_negotiation.cc:407] Connection header received
0811 11:51:14.069989 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:14.069992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:14.070049 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:14.070126 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:14.071539 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.072090 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.073286 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.073504 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.073597 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:14.076359 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:51:14.076362 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:51:14.076365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:51:14.076393 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:51:14.077930 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:51:14.078486 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:51:14.078489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:51:14.078490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:51:14.078576 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:51:14.078914 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:51:14.078917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:51:14.078918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:51:14.079113 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:51:14.079200 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 11:51:14.079737 (+ 537us) server_negotiation.cc:299] Negotiation successful
0811 11:51:14.079873 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260811 11:51:14.080932 21588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:14.067465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:33093 (local address 127.3.221.67:47195)
0811 11:51:14.067985 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:14.068070 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:14.069755 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:14.070401 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:14.070415 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:14.070831 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:51:14.071361 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:14.071378 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.072215 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.072218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:14.073119 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:14.073130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.073658 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.073662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:14.073843 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:14.074449 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:51:14.074468 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:51:14.076141 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:51:14.078059 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:51:14.078066 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:51:14.078088 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:51:14.078371 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:51:14.078661 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:51:14.078664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:51:14.078667 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:51:14.078818 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:51:14.079214 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:51:14.079222 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:51:14.079494 (+ 272us) client_negotiation.cc:769] Sending connection context
0811 11:51:14.079673 (+ 179us) client_negotiation.cc:240] Negotiation successful
0811 11:51:14.080031 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":126,"threads_started":1}
I20260811 11:51:14.082172 21585 heartbeater.cc:347] Connected to a master server at 127.3.221.126:33093
I20260811 11:51:14.082433 21585 heartbeater.cc:464] Registering TS with master...
I20260811 11:51:14.083036 21585 heartbeater.cc:511] Master 127.3.221.126:33093 requested a full tablet report, sending...
I20260811 11:51:14.084064 20883 ts_manager.cc:194] Registered new tserver with Master: 697c879c27354deba3f695e21716ab2d (127.3.221.67:42547)
I20260811 11:51:14.084613 20883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:47195
I20260811 11:51:14.094816 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:51:14.103799 20942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:14.096777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.3.221.126:33093)
0811 11:51:14.096919 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:14.096924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:14.097094 (+ 170us) server_negotiation.cc:407] Connection header received
0811 11:51:14.097162 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:14.097164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:14.097254 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:14.097345 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:51:14.098014 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.098527 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.099468 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.099672 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.099747 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:14.101067 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:51:14.101070 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:51:14.101072 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:51:14.101098 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:51:14.102311 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:51:14.102716 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:51:14.102719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:51:14.102721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:51:14.102812 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:51:14.103031 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:51:14.103033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:51:14.103034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:51:14.103250 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:51:14.103356 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 11:51:14.103542 (+ 186us) server_negotiation.cc:299] Negotiation successful
0811 11:51:14.103640 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 11:51:14.106736 20883 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:51:14.109856 20883 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:51:14.125020 21604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:14.118938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42756 (local address 127.3.221.67:42547)
0811 11:51:14.120425 (+ 1487us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:14.120436 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:14.120454 (+ 18us) server_negotiation.cc:407] Connection header received
0811 11:51:14.120528 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:14.120533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:14.120732 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:14.121209 (+ 477us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:51:14.121389 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.122101 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.123517 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.124355 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.124466 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:14.124616 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0811 11:51:14.124749 (+ 133us) server_negotiation.cc:299] Negotiation successful
0811 11:51:14.124854 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1393,"thread_start_us":114,"threads_started":1}
I20260811 11:51:14.125015 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:14.118808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:42547 (local address 127.0.0.1:42756)
0811 11:51:14.119921 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:14.119985 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:14.120124 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:14.120892 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:14.120895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:14.120913 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:51:14.121054 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:14.121061 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.122231 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.122234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:14.123397 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:14.123408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.124482 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.124485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:14.124573 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:14.124606 (+ 33us) client_negotiation.cc:769] Sending connection context
0811 11:51:14.124720 (+ 114us) client_negotiation.cc:240] Negotiation successful
0811 11:51:14.124824 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"spinlock_wait_cycles":12672,"thread_start_us":119,"threads_started":1}
I20260811 11:51:14.126494 21603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:14.119092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:36483 (local address 127.0.0.1:47112)
0811 11:51:14.120025 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:14.120036 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:14.120123 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:14.120855 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:14.120860 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:14.120884 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:51:14.121042 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:14.121050 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.122599 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.122602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:14.123691 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:14.123702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.126089 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.126093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:14.126180 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:14.126207 (+ 27us) client_negotiation.cc:769] Sending connection context
0811 11:51:14.126275 (+ 68us) client_negotiation.cc:240] Negotiation successful
0811 11:51:14.126349 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"mutex_wait_us":33,"thread_start_us":83,"threads_started":1}
I20260811 11:51:14.126551 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:14.118867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:35291 (local address 127.0.0.1:52788)
0811 11:51:14.119919 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:14.119977 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:14.120123 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:14.121052 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:14.121055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:14.121071 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:51:14.121156 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:14.121161 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.122638 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.122640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:14.124742 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:14.124753 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.126140 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.126143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:14.126244 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:14.126268 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 11:51:14.126340 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 11:51:14.126416 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"spinlock_wait_cycles":2304,"thread_start_us":121,"threads_started":1}
I20260811 11:51:14.126981 21610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:14.119462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52788 (local address 127.3.221.65:35291)
0811 11:51:14.120412 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:14.120428 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:14.120452 (+ 24us) server_negotiation.cc:407] Connection header received
0811 11:51:14.120528 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:14.120533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:14.120732 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:14.120884 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:51:14.121311 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.122476 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.124881 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.125822 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.125924 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:14.126072 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0811 11:51:14.126727 (+ 655us) server_negotiation.cc:299] Negotiation successful
0811 11:51:14.126833 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":838,"thread_start_us":85,"threads_started":1}
I20260811 11:51:14.127030 21611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:14.119673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47112 (local address 127.3.221.66:36483)
0811 11:51:14.120442 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:14.120454 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:14.120478 (+ 24us) server_negotiation.cc:407] Connection header received
0811 11:51:14.120545 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:14.120548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:14.120707 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:14.120847 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:51:14.121167 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.122429 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.124905 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.125732 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.125848 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:14.126002 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0811 11:51:14.126785 (+ 783us) server_negotiation.cc:299] Negotiation successful
0811 11:51:14.126889 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":150,"threads_started":1}
I20260811 11:51:14.128444 21494 tablet_service.cc:1467] Processing CreateTablet for tablet b2b34ed2e20c4ca395e50d4e4d639e49 (DEFAULT_TABLE table=test-table [id=52594c74a4564baa87a68a737f912784]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:51:14.129635 21494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b34ed2e20c4ca395e50d4e4d639e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:51:14.132058 21078 tablet_service.cc:1467] Processing CreateTablet for tablet b2b34ed2e20c4ca395e50d4e4d639e49 (DEFAULT_TABLE table=test-table [id=52594c74a4564baa87a68a737f912784]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:51:14.132153 21286 tablet_service.cc:1467] Processing CreateTablet for tablet b2b34ed2e20c4ca395e50d4e4d639e49 (DEFAULT_TABLE table=test-table [id=52594c74a4564baa87a68a737f912784]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:51:14.133049 21078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b34ed2e20c4ca395e50d4e4d639e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:51:14.133278 21286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b34ed2e20c4ca395e50d4e4d639e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:51:14.138029 21616 tablet_bootstrap.cc:492] T b2b34ed2e20c4ca395e50d4e4d639e49 P 697c879c27354deba3f695e21716ab2d: Bootstrap starting.
I20260811 11:51:14.139578 21617 tablet_bootstrap.cc:492] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4: Bootstrap starting.
I20260811 11:51:14.140583 21618 tablet_bootstrap.cc:492] T b2b34ed2e20c4ca395e50d4e4d639e49 P 91e49a8e21874a26953d0768d1eb84d1: Bootstrap starting.
I20260811 11:51:14.141119 21616 tablet_bootstrap.cc:654] T b2b34ed2e20c4ca395e50d4e4d639e49 P 697c879c27354deba3f695e21716ab2d: Neither blocks nor log segments found. Creating new log.
I20260811 11:51:14.142050 21617 tablet_bootstrap.cc:654] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4: Neither blocks nor log segments found. Creating new log.
I20260811 11:51:14.142746 21617 log.cc:824] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4: Log is configured to *not* fsync() on all Append() calls
I20260811 11:51:14.142719 21616 log.cc:824] T b2b34ed2e20c4ca395e50d4e4d639e49 P 697c879c27354deba3f695e21716ab2d: Log is configured to *not* fsync() on all Append() calls
I20260811 11:51:14.143642 21618 tablet_bootstrap.cc:654] T b2b34ed2e20c4ca395e50d4e4d639e49 P 91e49a8e21874a26953d0768d1eb84d1: Neither blocks nor log segments found. Creating new log.
I20260811 11:51:14.144316 21618 log.cc:824] T b2b34ed2e20c4ca395e50d4e4d639e49 P 91e49a8e21874a26953d0768d1eb84d1: Log is configured to *not* fsync() on all Append() calls
I20260811 11:51:14.144860 21616 tablet_bootstrap.cc:492] T b2b34ed2e20c4ca395e50d4e4d639e49 P 697c879c27354deba3f695e21716ab2d: No bootstrap required, opened a new log
I20260811 11:51:14.144860 21617 tablet_bootstrap.cc:492] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4: No bootstrap required, opened a new log
I20260811 11:51:14.145079 21617 ts_tablet_manager.cc:1397] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 11:51:14.145079 21616 ts_tablet_manager.cc:1397] T b2b34ed2e20c4ca395e50d4e4d639e49 P 697c879c27354deba3f695e21716ab2d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260811 11:51:14.146073 21618 tablet_bootstrap.cc:492] T b2b34ed2e20c4ca395e50d4e4d639e49 P 91e49a8e21874a26953d0768d1eb84d1: No bootstrap required, opened a new log
I20260811 11:51:14.146260 21618 ts_tablet_manager.cc:1397] T b2b34ed2e20c4ca395e50d4e4d639e49 P 91e49a8e21874a26953d0768d1eb84d1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 11:51:14.149195 21616 raft_consensus.cc:348] T b2b34ed2e20c4ca395e50d4e4d639e49 P 697c879c27354deba3f695e21716ab2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697c879c27354deba3f695e21716ab2d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 42547 } } peers { permanent_uuid: "e6fc6e7f76c7454caf2912ab46c11bb4" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35291 } } peers { permanent_uuid: "91e49a8e21874a26953d0768d1eb84d1" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 36483 } }
I20260811 11:51:14.149369 21618 raft_consensus.cc:348] T b2b34ed2e20c4ca395e50d4e4d639e49 P 91e49a8e21874a26953d0768d1eb84d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697c879c27354deba3f695e21716ab2d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 42547 } } peers { permanent_uuid: "e6fc6e7f76c7454caf2912ab46c11bb4" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35291 } } peers { permanent_uuid: "91e49a8e21874a26953d0768d1eb84d1" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 36483 } }
I20260811 11:51:14.149556 21616 raft_consensus.cc:374] T b2b34ed2e20c4ca395e50d4e4d639e49 P 697c879c27354deba3f695e21716ab2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:51:14.149591 21618 raft_consensus.cc:374] T b2b34ed2e20c4ca395e50d4e4d639e49 P 91e49a8e21874a26953d0768d1eb84d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:51:14.149622 21616 raft_consensus.cc:729] T b2b34ed2e20c4ca395e50d4e4d639e49 P 697c879c27354deba3f695e21716ab2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 697c879c27354deba3f695e21716ab2d, State: Initialized, Role: FOLLOWER
I20260811 11:51:14.149638 21618 raft_consensus.cc:729] T b2b34ed2e20c4ca395e50d4e4d639e49 P 91e49a8e21874a26953d0768d1eb84d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91e49a8e21874a26953d0768d1eb84d1, State: Initialized, Role: FOLLOWER
I20260811 11:51:14.149557 21617 raft_consensus.cc:348] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697c879c27354deba3f695e21716ab2d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 42547 } } peers { permanent_uuid: "e6fc6e7f76c7454caf2912ab46c11bb4" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35291 } } peers { permanent_uuid: "91e49a8e21874a26953d0768d1eb84d1" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 36483 } }
I20260811 11:51:14.149793 21617 raft_consensus.cc:374] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:51:14.149848 21617 raft_consensus.cc:729] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6fc6e7f76c7454caf2912ab46c11bb4, State: Initialized, Role: FOLLOWER
I20260811 11:51:14.150106 21618 consensus_queue.cc:260] T b2b34ed2e20c4ca395e50d4e4d639e49 P 91e49a8e21874a26953d0768d1eb84d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697c879c27354deba3f695e21716ab2d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 42547 } } peers { permanent_uuid: "e6fc6e7f76c7454caf2912ab46c11bb4" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35291 } } peers { permanent_uuid: "91e49a8e21874a26953d0768d1eb84d1" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 36483 } }
I20260811 11:51:14.150106 21616 consensus_queue.cc:260] T b2b34ed2e20c4ca395e50d4e4d639e49 P 697c879c27354deba3f695e21716ab2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697c879c27354deba3f695e21716ab2d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 42547 } } peers { permanent_uuid: "e6fc6e7f76c7454caf2912ab46c11bb4" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35291 } } peers { permanent_uuid: "91e49a8e21874a26953d0768d1eb84d1" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 36483 } }
I20260811 11:51:14.150193 21617 consensus_queue.cc:260] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697c879c27354deba3f695e21716ab2d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 42547 } } peers { permanent_uuid: "e6fc6e7f76c7454caf2912ab46c11bb4" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35291 } } peers { permanent_uuid: "91e49a8e21874a26953d0768d1eb84d1" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 36483 } }
I20260811 11:51:14.150803 21396 heartbeater.cc:503] Master 127.3.221.126:33093 was elected leader, sending a full tablet report...
I20260811 11:51:14.151103 21618 ts_tablet_manager.cc:1428] T b2b34ed2e20c4ca395e50d4e4d639e49 P 91e49a8e21874a26953d0768d1eb84d1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260811 11:51:14.151144 21616 ts_tablet_manager.cc:1428] T b2b34ed2e20c4ca395e50d4e4d639e49 P 697c879c27354deba3f695e21716ab2d: Time spent starting tablet: real 0.006s user 0.001s sys 0.006s
I20260811 11:51:14.151504 21585 heartbeater.cc:503] Master 127.3.221.126:33093 was elected leader, sending a full tablet report...
I20260811 11:51:14.151692 21617 ts_tablet_manager.cc:1428] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260811 11:51:14.152199 21178 heartbeater.cc:503] Master 127.3.221.126:33093 was elected leader, sending a full tablet report...
I20260811 11:51:14.275863 21629 raft_consensus.cc:482] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:51:14.276085 21629 raft_consensus.cc:504] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697c879c27354deba3f695e21716ab2d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 42547 } } peers { permanent_uuid: "e6fc6e7f76c7454caf2912ab46c11bb4" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35291 } } peers { permanent_uuid: "91e49a8e21874a26953d0768d1eb84d1" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 36483 } }
I20260811 11:51:14.277205 21629 leader_election.cc:288] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 697c879c27354deba3f695e21716ab2d (127.3.221.67:42547), 91e49a8e21874a26953d0768d1eb84d1 (127.3.221.66:36483)
I20260811 11:51:14.281287 21611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:14.277724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:56833 (local address 127.3.221.66:36483)
0811 11:51:14.277834 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:14.277838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:14.278230 (+ 392us) server_negotiation.cc:407] Connection header received
0811 11:51:14.278346 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:14.278349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:14.278394 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:14.278467 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:51:14.278728 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.279214 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.280283 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.280703 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.280818 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:14.280876 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 11:51:14.281063 (+ 187us) server_negotiation.cc:299] Negotiation successful
0811 11:51:14.281133 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 11:51:14.281333 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:14.277546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:36483 (local address 127.3.221.65:56833)
0811 11:51:14.278111 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:14.278125 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:14.278239 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:14.278479 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:14.278482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:14.278498 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:51:14.278622 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:14.278627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.279321 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.279324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:14.280158 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:14.280169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.280822 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.280825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:14.280936 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:14.280965 (+ 29us) client_negotiation.cc:769] Sending connection context
0811 11:51:14.281037 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 11:51:14.281113 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":34,"thread_start_us":93,"threads_started":1}
I20260811 11:51:14.281354 21604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:14.277636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:53199 (local address 127.3.221.67:42547)
0811 11:51:14.277790 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:14.277795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:14.278086 (+ 291us) server_negotiation.cc:407] Connection header received
0811 11:51:14.278210 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:14.278213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:14.278260 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:14.278325 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:51:14.278589 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.279096 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.280282 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.280706 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.280818 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:14.280876 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 11:51:14.281131 (+ 255us) server_negotiation.cc:299] Negotiation successful
0811 11:51:14.281207 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 11:51:14.281371 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:14.277546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:42547 (local address 127.3.221.65:53199)
0811 11:51:14.277978 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:14.277990 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:51:14.278116 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:51:14.278353 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:51:14.278356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:51:14.278372 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:51:14.278486 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:14.278492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.279202 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.279205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:14.280156 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:51:14.280167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.280830 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.280833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:51:14.280974 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:14.280997 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 11:51:14.281106 (+ 109us) client_negotiation.cc:240] Negotiation successful
0811 11:51:14.281180 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":20,"thread_start_us":132,"threads_started":1}
I20260811 11:51:14.282119 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b34ed2e20c4ca395e50d4e4d639e49" candidate_uuid: "e6fc6e7f76c7454caf2912ab46c11bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91e49a8e21874a26953d0768d1eb84d1" is_pre_election: true
I20260811 11:51:14.282121 21524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b34ed2e20c4ca395e50d4e4d639e49" candidate_uuid: "e6fc6e7f76c7454caf2912ab46c11bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "697c879c27354deba3f695e21716ab2d" is_pre_election: true
I20260811 11:51:14.282431 21324 raft_consensus.cc:2454] T b2b34ed2e20c4ca395e50d4e4d639e49 P 91e49a8e21874a26953d0768d1eb84d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6fc6e7f76c7454caf2912ab46c11bb4 in term 0.
I20260811 11:51:14.282435 21524 raft_consensus.cc:2454] T b2b34ed2e20c4ca395e50d4e4d639e49 P 697c879c27354deba3f695e21716ab2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6fc6e7f76c7454caf2912ab46c11bb4 in term 0.
I20260811 11:51:14.282919 21003 leader_election.cc:302] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 697c879c27354deba3f695e21716ab2d, e6fc6e7f76c7454caf2912ab46c11bb4; no voters:
I20260811 11:51:14.283200 21629 raft_consensus.cc:2781] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:51:14.283293 21629 raft_consensus.cc:482] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:51:14.283352 21629 raft_consensus.cc:3037] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:51:14.284395 21629 raft_consensus.cc:504] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697c879c27354deba3f695e21716ab2d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 42547 } } peers { permanent_uuid: "e6fc6e7f76c7454caf2912ab46c11bb4" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35291 } } peers { permanent_uuid: "91e49a8e21874a26953d0768d1eb84d1" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 36483 } }
I20260811 11:51:14.284734 21629 leader_election.cc:288] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4 [CANDIDATE]: Term 1 election: Requested vote from peers 697c879c27354deba3f695e21716ab2d (127.3.221.67:42547), 91e49a8e21874a26953d0768d1eb84d1 (127.3.221.66:36483)
I20260811 11:51:14.285130 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b34ed2e20c4ca395e50d4e4d639e49" candidate_uuid: "e6fc6e7f76c7454caf2912ab46c11bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91e49a8e21874a26953d0768d1eb84d1"
I20260811 11:51:14.285127 21524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b34ed2e20c4ca395e50d4e4d639e49" candidate_uuid: "e6fc6e7f76c7454caf2912ab46c11bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "697c879c27354deba3f695e21716ab2d"
I20260811 11:51:14.285280 21524 raft_consensus.cc:3037] T b2b34ed2e20c4ca395e50d4e4d639e49 P 697c879c27354deba3f695e21716ab2d [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:51:14.285270 21324 raft_consensus.cc:3037] T b2b34ed2e20c4ca395e50d4e4d639e49 P 91e49a8e21874a26953d0768d1eb84d1 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:51:14.286304 21324 raft_consensus.cc:2454] T b2b34ed2e20c4ca395e50d4e4d639e49 P 91e49a8e21874a26953d0768d1eb84d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6fc6e7f76c7454caf2912ab46c11bb4 in term 1.
I20260811 11:51:14.286311 21524 raft_consensus.cc:2454] T b2b34ed2e20c4ca395e50d4e4d639e49 P 697c879c27354deba3f695e21716ab2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6fc6e7f76c7454caf2912ab46c11bb4 in term 1.
I20260811 11:51:14.286597 21001 leader_election.cc:302] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91e49a8e21874a26953d0768d1eb84d1, e6fc6e7f76c7454caf2912ab46c11bb4; no voters:
I20260811 11:51:14.286775 21629 raft_consensus.cc:2781] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:51:14.287025 21629 raft_consensus.cc:686] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4 [term 1 LEADER]: Becoming Leader. State: Replica: e6fc6e7f76c7454caf2912ab46c11bb4, State: Running, Role: LEADER
I20260811 11:51:14.287396 21629 consensus_queue.cc:237] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697c879c27354deba3f695e21716ab2d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 42547 } } peers { permanent_uuid: "e6fc6e7f76c7454caf2912ab46c11bb4" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35291 } } peers { permanent_uuid: "91e49a8e21874a26953d0768d1eb84d1" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 36483 } }
I20260811 11:51:14.290807 20883 catalog_manager.cc:5495] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6fc6e7f76c7454caf2912ab46c11bb4 (127.3.221.65). New cstate: current_term: 1 leader_uuid: "e6fc6e7f76c7454caf2912ab46c11bb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697c879c27354deba3f695e21716ab2d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 42547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6fc6e7f76c7454caf2912ab46c11bb4" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 35291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91e49a8e21874a26953d0768d1eb84d1" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 36483 } health_report { overall_health: UNKNOWN } } }
W20260811 11:51:14.316088 21586 tablet.cc:2369] T b2b34ed2e20c4ca395e50d4e4d639e49 P 697c879c27354deba3f695e21716ab2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:51:14.316998 21610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:14.313115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52798 (local address 127.3.221.65:35291)
0811 11:51:14.313326 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0811 11:51:14.313330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:51:14.313345 (+ 15us) server_negotiation.cc:407] Connection header received
0811 11:51:14.313427 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:51:14.313430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:51:14.313489 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:51:14.313595 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:51:14.313926 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.314440 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.315412 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:51:14.315654 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:51:14.315757 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:51:14.316080 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:51:14.316606 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:51:14.316700 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0811 11:51:14.316794 (+ 94us) server_negotiation.cc:299] Negotiation successful
0811 11:51:14.316855 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260811 11:51:14.323143 21524 raft_consensus.cc:1261] T b2b34ed2e20c4ca395e50d4e4d639e49 P 697c879c27354deba3f695e21716ab2d [term 1 FOLLOWER]: Refusing update from remote peer e6fc6e7f76c7454caf2912ab46c11bb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:51:14.323169 21324 raft_consensus.cc:1261] T b2b34ed2e20c4ca395e50d4e4d639e49 P 91e49a8e21874a26953d0768d1eb84d1 [term 1 FOLLOWER]: Refusing update from remote peer e6fc6e7f76c7454caf2912ab46c11bb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:51:14.323995 21635 consensus_queue.cc:1035] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "697c879c27354deba3f695e21716ab2d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 42547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:51:14.324201 21629 consensus_queue.cc:1035] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91e49a8e21874a26953d0768d1eb84d1" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 36483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:51:14.333768 21642 mvcc.cc:205] Tried to move back new op lower bound from 7317295408420048896 to 7317295408284114944. Current Snapshot: MvccSnapshot[applied={T|T < 7317295408420048896}]
I20260811 11:51:14.342803 20883 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:51:14.343029 20883 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:51:14.345870 20883 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6d1c0b94e574afb838e14b324e211a6: Sending DeleteTablet for 3 replicas of tablet b2b34ed2e20c4ca395e50d4e4d639e49
I20260811 11:51:14.346700 21286 tablet_service.cc:1514] Processing DeleteTablet for tablet b2b34ed2e20c4ca395e50d4e4d639e49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:51:14 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:47112
I20260811 11:51:14.346722 21494 tablet_service.cc:1514] Processing DeleteTablet for tablet b2b34ed2e20c4ca395e50d4e4d639e49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:51:14 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:42756
I20260811 11:51:14.346711 21078 tablet_service.cc:1514] Processing DeleteTablet for tablet b2b34ed2e20c4ca395e50d4e4d639e49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:51:14 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:52788
I20260811 11:51:14.347259 21657 tablet_replica.cc:330] T b2b34ed2e20c4ca395e50d4e4d639e49 P 697c879c27354deba3f695e21716ab2d: stopping tablet replica
I20260811 11:51:14.347302 21661 tablet_replica.cc:330] T b2b34ed2e20c4ca395e50d4e4d639e49 P 91e49a8e21874a26953d0768d1eb84d1: stopping tablet replica
I20260811 11:51:14.347657 21661 raft_consensus.cc:2229] T b2b34ed2e20c4ca395e50d4e4d639e49 P 91e49a8e21874a26953d0768d1eb84d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:51:14.347667 21657 raft_consensus.cc:2229] T b2b34ed2e20c4ca395e50d4e4d639e49 P 697c879c27354deba3f695e21716ab2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:51:14.348034 21661 raft_consensus.cc:2258] T b2b34ed2e20c4ca395e50d4e4d639e49 P 91e49a8e21874a26953d0768d1eb84d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:51:14.348083 21657 raft_consensus.cc:2258] T b2b34ed2e20c4ca395e50d4e4d639e49 P 697c879c27354deba3f695e21716ab2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:51:14.348235 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 20974
I20260811 11:51:14.348390 21660 tablet_replica.cc:330] T b2b34ed2e20c4ca395e50d4e4d639e49 P e6fc6e7f76c7454caf2912ab46c11bb4: stopping tablet replica
I20260811 11:51:14.352303 21657 ts_tablet_manager.cc:1905] T b2b34ed2e20c4ca395e50d4e4d639e49 P 697c879c27354deba3f695e21716ab2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:51:14.355343 21657 ts_tablet_manager.cc:1918] T b2b34ed2e20c4ca395e50d4e4d639e49 P 697c879c27354deba3f695e21716ab2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:51:14.355674 21657 log.cc:1196] T b2b34ed2e20c4ca395e50d4e4d639e49 P 697c879c27354deba3f695e21716ab2d: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/b2b34ed2e20c4ca395e50d4e4d639e49
I20260811 11:51:14.356119 21657 ts_tablet_manager.cc:1939] T b2b34ed2e20c4ca395e50d4e4d639e49 P 697c879c27354deba3f695e21716ab2d: Deleting consensus metadata
I20260811 11:51:14.357352 21661 ts_tablet_manager.cc:1905] T b2b34ed2e20c4ca395e50d4e4d639e49 P 91e49a8e21874a26953d0768d1eb84d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:51:14.357360 20857 catalog_manager.cc:4841] TS 697c879c27354deba3f695e21716ab2d (127.3.221.67:42547): tablet b2b34ed2e20c4ca395e50d4e4d639e49 (table test-table [id=52594c74a4564baa87a68a737f912784]) successfully deleted
I20260811 11:51:14.360126 21661 ts_tablet_manager.cc:1918] T b2b34ed2e20c4ca395e50d4e4d639e49 P 91e49a8e21874a26953d0768d1eb84d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:51:14.360242 21661 log.cc:1196] T b2b34ed2e20c4ca395e50d4e4d639e49 P 91e49a8e21874a26953d0768d1eb84d1: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/b2b34ed2e20c4ca395e50d4e4d639e49
I20260811 11:51:14.360567 21661 ts_tablet_manager.cc:1939] T b2b34ed2e20c4ca395e50d4e4d639e49 P 91e49a8e21874a26953d0768d1eb84d1: Deleting consensus metadata
I20260811 11:51:14.362042 20856 catalog_manager.cc:4841] TS 91e49a8e21874a26953d0768d1eb84d1 (127.3.221.66:36483): tablet b2b34ed2e20c4ca395e50d4e4d639e49 (table test-table [id=52594c74a4564baa87a68a737f912784]) successfully deleted
W20260811 11:51:14.363654 20860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.221.65:35291 (error 108)
I20260811 11:51:14.364508 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 21186
I20260811 11:51:14.364835 21603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:51:14.364527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:35291 (local address 127.0.0.1:52806)
0811 11:51:14.364666 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0811 11:51:14.364736 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.65:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260811 11:51:14.384186 20860 catalog_manager.cc:4584] TS e6fc6e7f76c7454caf2912ab46c11bb4 (127.3.221.65:35291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b2b34ed2e20c4ca395e50d4e4d639e49: Network error: Client connection negotiation failed: client connection to 127.3.221.65:35291: connect: Connection refused (error 111)
I20260811 11:51:14.390038 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 21403
I20260811 11:51:14.404524 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 20834
2026-08-11T11:51:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2604 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:51:14 dist-test-slave-bndk krb5kdc[21667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:51:14 dist-test-slave-bndk krb5kdc[21667](info): set up 2 sockets
Aug 11 11:51:14 dist-test-slave-bndk krb5kdc[21667](info): commencing operation
krb5kdc: starting...
W20260811 11:51:15.585806 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:51:15 dist-test-slave-bndk krb5kdc[21667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:51:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:51:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/rangeradmin_127.3.221.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/rangeradmin_127.3.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/rangerlookup_127.3.221.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/rangerlookup_127.3.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/HTTP_127.3.221.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/HTTP_127.3.221.84@KRBTEST.COM.keytab.
I20260811 11:51:15.766173 3957 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskt5cui_/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/postgres -l logfile start
2026-08-11 11:51:18.369 UTC [21710] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-11 11:51:18.369 UTC [21710] LOG: listening on IPv4 address "127.3.221.84", port 39021
2026-08-11 11:51:18.373 UTC [21710] LOG: listening on Unix socket "/tmp/.s.PGSQL.39021"
2026-08-11 11:51:18.395 UTC [21715] LOG: database system was shut down at 2026-08-11 11:51:16 UTC
2026-08-11 11:51:18.399 UTC [21710] LOG: database system is ready to accept connections
I20260811 11:51:19.469236 3957 mini_postgres.cc:96] Postgres bound to 39021
2026-08-11 11:51:19.477 UTC [21725] FATAL: database "slave" does not exist
127.3.221.84:39021 - accepting connections
I20260811 11:51:19.478430 3957 mini_ranger.cc:161] Starting Ranger...
I20260811 11:51:19.495642 3957 mini_ranger.cc:84] Created miniranger Postgres user
I20260811 11:51:20.021178 3957 mini_ranger.cc:87] Created ranger Postgres database
I20260811 11:51:20.021337 3957 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ranger-admin
2026-08-11 11:51:20,353 [I] DB FLAVOR :POSTGRES
2026-08-11 11:51:20,353 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 11:51:20,353 [I] Checking connection..
2026-08-11 11:51:20,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-11 11:51:20,612 [I] Checking connection passed.
2026-08-11 11:51:20,613 [I] --------- Verifying version history table ---------
2026-08-11 11:51:20,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 11:51:20,882 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 11:51:20,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 11:51:21,145 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 11:51:21,145 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-11 11:51:21,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-11 11:51:21.411 UTC [21840] WARNING: there is no transaction in progress
2026-08-11 11:51:21,420 [I] create_dbversion_catalog.sql file imported successfully
2026-08-11 11:51:21,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 11:51:21,675 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-11 11:51:21,675 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 11:51:21,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-11 11:51:21,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-11 11:51:22,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-bndk.c.gcp-upstream.internal','N') ;"
2026-08-11 11:51:22,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 11:51:22,698 [I] Table x_portal_user does not exist in database ranger
2026-08-11 11:51:22,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 11:51:22,949 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-11 11:51:22,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 11:51:23,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 11:51:23,480 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-11 11:51:23,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-11 11:51:24.507 UTC [22012] WARNING: there is no transaction in progress
2026-08-11 11:51:24.524 UTC [22012] WARNING: there is no transaction in progress
2026-08-11 11:51:24.542 UTC [22012] WARNING: there is no transaction in progress
2026-08-11 11:51:24.558 UTC [22012] WARNING: there is no transaction in progress
2026-08-11 11:51:24.574 UTC [22012] WARNING: there is no transaction in progress
2026-08-11 11:51:24.596 UTC [22012] WARNING: there is no transaction in progress
2026-08-11 11:51:24.663 UTC [22012] WARNING: there is no transaction in progress
2026-08-11 11:51:24.670 UTC [22012] WARNING: there is no transaction in progress
2026-08-11 11:51:24.685 UTC [22012] WARNING: there is no transaction in progress
2026-08-11 11:51:24.695 UTC [22012] WARNING: there is no transaction in progress
2026-08-11 11:51:24.705 UTC [22012] WARNING: there is no transaction in progress
2026-08-11 11:51:24.719 UTC [22012] WARNING: there is no transaction in progress
2026-08-11 11:51:24.729 UTC [22012] WARNING: there is no transaction in progress
2026-08-11 11:51:24.736 UTC [22012] WARNING: there is no transaction in progress
2026-08-11 11:51:24.743 UTC [22012] WARNING: there is no transaction in progress
2026-08-11 11:51:24.750 UTC [22012] WARNING: there is no transaction in progress
2026-08-11 11:51:25,420 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-11 11:51:25,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-11 11:51:25,680 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-11 11:51:25,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 11:51:25,928 [I] Table x_portal_user already exists in database 'ranger'
2026-08-11 11:51:25,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 11:51:26,189 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-11 11:51:26,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 11:51:26,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 11:51:26,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-bndk.c.gcp-upstream.internal';"
2026-08-11 11:51:26,963 [I] CORE_DB_SCHEMA import status has been updated
2026-08-11 11:51:26,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-11 11:51:27,239 [I] DB_PATCHES have already been applied
I20260811 11:51:27.244709 3957 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ranger-admin:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskt5cui_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260811 11:51:27.244781 3957 mini_ranger.cc:193] Using host: 127.3.221.84
I20260811 11:51:27.249646 3957 mini_ranger.cc:227] Ranger admin URL: http://127.3.221.84:54513
Aug 11, 2026 11:51:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 11:51:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 11:51:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ranger-admin/ews/webapp
Aug 11, 2026 11:51:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 11, 2026 11:51:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 11, 2026 11:51:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ranger-admin/ews/webapp].
Aug 11, 2026 11:51:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260811 11:51:28.544154 3957 mini_ranger.cc:160] Time spent starting Ranger: real 9.066s user 0.004s sys 0.008s
Aug 11, 2026 11:51:49 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 11, 2026 11:51:49 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 11, 2026 11:51:49 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 11, 2026 11:51:49 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 11, 2026 11:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 11, 2026 11:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 11, 2026 11:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 11, 2026 11:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 11, 2026 11:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 11, 2026 11:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 11, 2026 11:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 11, 2026 11:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 11, 2026 11:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 11, 2026 11:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 11, 2026 11:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 11, 2026 11:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 11, 2026 11:51:49 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 11, 2026 11:51:50 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260811 11:51:51.079764 3957 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/rangerkms_127.3.221.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/rangerkms_127.3.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.221.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.221.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.221.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/HTTP_127.3.221.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.221.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/HTTP_127.3.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 11 11:51:51 dist-test-slave-bndk krb5kdc[21667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449111, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260811 11:51:51.199785 3957 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260811 11:51:51.212961 3957 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260811 11:51:52.194459 3957 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260811 11:51:52.194566 3957 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ranger-kms
I20260811 11:51:52.194693 3957 mini_ranger_kms.cc:227] Using postgres at 127.3.221.84:39021
2026-08-11 11:51:52,881 [I] DB FLAVOR :POSTGRES
2026-08-11 11:51:52,882 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 11:51:52,882 [I] Checking connection
2026-08-11 11:51:52,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-11 11:51:53,150 [I] connection success
2026-08-11 11:51:53,151 [I] --------- Verifying Ranger DB tables ---------
2026-08-11 11:51:53,151 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-11 11:51:53,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-11 11:51:53,423 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-11 11:51:53,424 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 11:51:53,424 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-11 11:51:53,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:39021/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-11 11:51:53,700 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260811 11:51:54.272535 3957 mini_ranger_kms.cc:310] Created kms service
I20260811 11:51:54.508716 3957 mini_ranger_kms.cc:326] Created kudu user
I20260811 11:51:54.553320 3957 mini_ranger_kms.cc:343] Created rangerkms user
Aug 11 11:51:54 dist-test-slave-bndk krb5kdc[21667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449111, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
I20260811 11:51:54.981725 3957 mini_ranger_kms.cc:384] Added ranger policy
I20260811 11:51:54.981892 3957 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ranger-kms:/tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260811 11:51:54.981914 3957 mini_ranger_kms.cc:240] Using host: 127.3.221.84
I20260811 11:51:54.986815 3957 mini_ranger_kms.cc:276] Ranger KMS PID: 22307
I20260811 11:51:54.986974 3957 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.221.84:48403
Aug 11, 2026 11:51:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 11:51:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 11:51:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 11, 2026 11:51:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 11, 2026 11:51:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ranger-kms/ews/webapp].
Aug 11, 2026 11:51:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 11 11:51:57 dist-test-slave-bndk krb5kdc[21667](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786449117, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.221.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:51:57.572280 3957 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.373s user 0.045s sys 0.253s
I20260811 11:51:57.572409 3957 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260811 11:51:57.572531 3957 mini_ranger_kms.cc:401] 127.3.221.84:48403/kms/v1/keys
Aug 11 11:51:57 dist-test-slave-bndk krb5kdc[21667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449117, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.221.84@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
Aug 11 11:51:58 dist-test-slave-bndk krb5kdc[21667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449117, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.221.84@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
Aug 11 11:52:00 dist-test-slave-bndk krb5kdc[21667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:52:00.836680 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:38377
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:38377
--ranger_config_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.221.84:48403/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:52:00.991410 22381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:52:00.991725 22381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:52:00.991796 22381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:52:00.996089 22381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:52:00.996170 22381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:52:00.996197 22381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:52:00.996218 22381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:52:00.996248 22381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:52:01.002252 22381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.221.84:48403/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:38377
--ranger_config_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:38377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:52:01.003420 22381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:52:01.004508 22381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:52:01.014174 22394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:52:01.014127 22389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:52:01.014513 22381 server_base.cc:1034] running on GCE node
W20260811 11:52:01.014132 22391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:52:01.015260 22381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:52:01.016862 22381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:52:01.018072 22381 hybrid_clock.cc:648] HybridClock initialized: now 1786449121018048 us; error 37 us; skew 500 ppm
Aug 11 11:52:01 dist-test-slave-bndk krb5kdc[21667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449121, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:52:01.021809 22381 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:52:01.022872 22381 webserver.cc:460] Webserver started at http://127.3.221.126:37757/ using document root <none> and password file <none>
I20260811 11:52:01.023461 22381 fs_manager.cc:359] Metadata directory not provided
I20260811 11:52:01.023526 22381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:52:01.023758 22381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 11:52:01 dist-test-slave-bndk krb5kdc[21667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449121, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
I20260811 11:52:01.076695 22381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "7cff43c5ee314d4385188acc81124764"
format_stamp: "Formatted at 2026-08-11 11:52:01 on dist-test-slave-bndk"
server_key: "b6cd23e57afc820e1e1affc30bd122ca"
server_key_iv: "882ac41680b982f27f8c5f5b52003845"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:01.077462 22381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "7cff43c5ee314d4385188acc81124764"
format_stamp: "Formatted at 2026-08-11 11:52:01 on dist-test-slave-bndk"
server_key: "b6cd23e57afc820e1e1affc30bd122ca"
server_key_iv: "882ac41680b982f27f8c5f5b52003845"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:01.081712 22381 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 11:52:01.096938 22410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:01.098452 22381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:52:01.098583 22381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "7cff43c5ee314d4385188acc81124764"
format_stamp: "Formatted at 2026-08-11 11:52:01 on dist-test-slave-bndk"
server_key: "b6cd23e57afc820e1e1affc30bd122ca"
server_key_iv: "882ac41680b982f27f8c5f5b52003845"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:01.098737 22381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:52:01.130261 22381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:52:01.133387 22381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:52:01.133615 22381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:52:01.142899 22381 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:38377
I20260811 11:52:01.142933 22493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:38377 every 8 connection(s)
I20260811 11:52:01.144191 22381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:52:01.147979 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 22381
I20260811 11:52:01.148133 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:52:01.148460 3957 external_mini_cluster.cc:1428] Setting key 9ce709cf50d6a8243430d5e921fb08e0
I20260811 11:52:01.149034 22494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:52:01.158835 22494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764: Bootstrap starting.
I20260811 11:52:01.161828 22494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764: Neither blocks nor log segments found. Creating new log.
Aug 11 11:52:01 dist-test-slave-bndk krb5kdc[21667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:52:01.162933 22494 log.cc:824] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764: Log is configured to *not* fsync() on all Append() calls
I20260811 11:52:01.165743 22494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764: No bootstrap required, opened a new log
I20260811 11:52:01.170264 22494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cff43c5ee314d4385188acc81124764" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38377 } }
I20260811 11:52:01.170590 22494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:52:01.170686 22494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cff43c5ee314d4385188acc81124764, State: Initialized, Role: FOLLOWER
I20260811 11:52:01.171037 22500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:01.152222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53854 (local address 127.3.221.126:38377)
0811 11:52:01.152982 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:01.153001 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:01.153575 (+ 574us) server_negotiation.cc:407] Connection header received
0811 11:52:01.154451 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:01.154489 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:01.154994 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:01.155565 (+ 571us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:52:01.156501 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:01.158090 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:01.159281 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:01.159977 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:01.160150 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:01.163273 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:52:01.163281 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:52:01.163309 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:52:01.163349 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:52:01.166770 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:01.167618 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:01.167647 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:01.167655 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:01.167861 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:01.168380 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:01.168383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:01.168385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:01.169021 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:52:01.169283 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:01.169613 (+ 330us) server_negotiation.cc:299] Negotiation successful
0811 11:52:01.169942 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":190,"threads_started":1}
I20260811 11:52:01.171262 22494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cff43c5ee314d4385188acc81124764" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38377 } }
I20260811 11:52:01.171461 22494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:52:01.171550 22494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:52:01.171666 22494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:52:01.172871 22494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cff43c5ee314d4385188acc81124764" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38377 } }
I20260811 11:52:01.173278 22494 leader_election.cc:302] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cff43c5ee314d4385188acc81124764; no voters:
I20260811 11:52:01.173655 22494 leader_election.cc:288] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:52:01.174081 22502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:52:01.174285 22502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764 [term 1 LEADER]: Becoming Leader. State: Replica: 7cff43c5ee314d4385188acc81124764, State: Running, Role: LEADER
I20260811 11:52:01.174686 22502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cff43c5ee314d4385188acc81124764" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38377 } }
I20260811 11:52:01.175091 22494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:52:01.176544 22505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cff43c5ee314d4385188acc81124764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cff43c5ee314d4385188acc81124764" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38377 } } }
I20260811 11:52:01.176687 22505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764 [sys.catalog]: This master's current role is: LEADER
I20260811 11:52:01.176927 22505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cff43c5ee314d4385188acc81124764. Latest consensus state: current_term: 1 leader_uuid: "7cff43c5ee314d4385188acc81124764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cff43c5ee314d4385188acc81124764" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38377 } } }
I20260811 11:52:01.177008 22505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764 [sys.catalog]: This master's current role is: LEADER
I20260811 11:52:01.177517 22521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:52:01.177793 22494 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260811 11:52:01.184552 22521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:52:01.194999 22521 catalog_manager.cc:1347] Generated new cluster ID: 5c99635722884f50b36279070d978f3d
I20260811 11:52:01.195147 22521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:52:01.226504 22521 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:52:01.227936 22521 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:52:01.240120 22521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764: Generated new TSK 0
I20260811 11:52:01.241199 22521 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
I20260811 11:52:02.001725 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.221.84:48403/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:38377
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:52:02.133139 22561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:52:02.133569 22561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:52:02.133642 22561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:52:02.138252 22561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:52:02.138322 22561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:52:02.138406 22561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:52:02.144452 22561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.221.84:48403/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:38377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:52:02.145658 22561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:52:02.146736 22561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:52:02.157449 22570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:52:02.157488 22572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:52:02.157476 22567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:52:02.157949 22561 server_base.cc:1034] running on GCE node
I20260811 11:52:02.158440 22561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:52:02.159132 22561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:52:02.160346 22561 hybrid_clock.cc:648] HybridClock initialized: now 1786449122160304 us; error 56 us; skew 500 ppm
Aug 11 11:52:02 dist-test-slave-bndk krb5kdc[21667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449122, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:52:02.163551 22561 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:52:02.164724 22561 webserver.cc:460] Webserver started at http://127.3.221.65:45165/ using document root <none> and password file <none>
I20260811 11:52:02.165583 22561 fs_manager.cc:359] Metadata directory not provided
I20260811 11:52:02.165751 22561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:52:02.166062 22561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 11:52:02 dist-test-slave-bndk krb5kdc[21667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449122, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
I20260811 11:52:02.186474 22561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "89c7a40cb720459ca96f1288a80a03a3"
format_stamp: "Formatted at 2026-08-11 11:52:02 on dist-test-slave-bndk"
server_key: "a944ec218af6917f8be5feb3d2fdfdd3"
server_key_iv: "ce3da23991275e623597a637586ec106"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:02.187465 22561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "89c7a40cb720459ca96f1288a80a03a3"
format_stamp: "Formatted at 2026-08-11 11:52:02 on dist-test-slave-bndk"
server_key: "a944ec218af6917f8be5feb3d2fdfdd3"
server_key_iv: "ce3da23991275e623597a637586ec106"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:02.194942 22561 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.004s
I20260811 11:52:02.218267 22589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:02.220021 22561 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.001s
I20260811 11:52:02.220182 22561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "89c7a40cb720459ca96f1288a80a03a3"
format_stamp: "Formatted at 2026-08-11 11:52:02 on dist-test-slave-bndk"
server_key: "a944ec218af6917f8be5feb3d2fdfdd3"
server_key_iv: "ce3da23991275e623597a637586ec106"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:02.220348 22561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:52:02.286906 22561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:52:02.290870 22561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:52:02.291168 22561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:52:02.292121 22561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:52:02.293382 22561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:52:02.293497 22561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:02.293591 22561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:52:02.293644 22561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:02.306540 22561 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:44655
I20260811 11:52:02.307996 22561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:52:02.313644 22769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:44655 every 8 connection(s)
I20260811 11:52:02.314276 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 22561
I20260811 11:52:02.314414 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:52:02.314785 3957 external_mini_cluster.cc:1428] Setting key 836ec60ba0dcbb55a1cfd499f8d7d7f9
Aug 11 11:52:02 dist-test-slave-bndk krb5kdc[21667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449122, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.221.66@KRBTEST.COM; defaulting to no policy
I20260811 11:52:02.334052 22774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:02.313343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:34141 (local address 127.3.221.126:38377)
0811 11:52:02.316488 (+ 3145us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:02.316493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:02.316510 (+ 17us) server_negotiation.cc:407] Connection header received
0811 11:52:02.317110 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:02.317114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:02.317171 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:02.317294 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:52:02.319104 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:02.319995 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:02.322467 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:02.322836 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:02.322952 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:02.328015 (+ 5063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:52:02.328020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:52:02.328023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:52:02.328063 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:52:02.330232 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:02.331130 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:02.331135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:02.331137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:02.331279 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:02.332230 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:02.332235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:02.332237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:02.332560 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:52:02.332673 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:02.333619 (+ 946us) server_negotiation.cc:299] Negotiation successful
0811 11:52:02.333787 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3055,"thread_start_us":77,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.66@KRBTEST.COM" created.
I20260811 11:52:02.335216 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:02.311947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:38377 (local address 127.3.221.65:34141)
0811 11:52:02.314573 (+ 2626us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:02.314641 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:02.316700 (+ 2059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:02.317667 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:52:02.317678 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:52:02.318104 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:52:02.318900 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:02.318925 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:02.320182 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:02.320186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:02.322263 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:02.322280 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:02.323177 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:02.323182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:02.323752 (+ 570us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:02.324987 (+ 1235us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:52:02.325031 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:52:02.327678 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:52:02.330439 (+ 2761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:02.330449 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:02.330475 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:02.330977 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:02.331508 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:02.331515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:02.331518 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:02.331984 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:02.332742 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:52:02.332753 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:52:02.333252 (+ 499us) client_negotiation.cc:769] Sending connection context
0811 11:52:02.333520 (+ 268us) client_negotiation.cc:240] Negotiation successful
0811 11:52:02.333961 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2351,"thread_start_us":200,"threads_started":1}
I20260811 11:52:02.338660 22770 heartbeater.cc:347] Connected to a master server at 127.3.221.126:38377
I20260811 11:52:02.338977 22770 heartbeater.cc:464] Registering TS with master...
I20260811 11:52:02.339663 22770 heartbeater.cc:511] Master 127.3.221.126:38377 requested a full tablet report, sending...
I20260811 11:52:02.341822 22439 ts_manager.cc:194] Registered new tserver with Master: 89c7a40cb720459ca96f1288a80a03a3 (127.3.221.65:44655)
I20260811 11:52:02.343478 22439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:34141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
I20260811 11:52:02.364955 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.221.84:48403/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:38377
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 11:52:02 dist-test-slave-bndk krb5kdc[21667](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786449122, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260811 11:52:02.528133 22779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:52:02.528664 22779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:52:02.528784 22779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:52:02.535439 22779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:52:02.535604 22779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:52:02.535727 22779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:52:02.544157 22779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.221.84:48403/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:38377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:52:02.545722 22779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:52:02.547241 22779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:52:02.560571 22799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:52:02.560727 22791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:52:02.560588 22790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:52:02.562182 22779 server_base.cc:1034] running on GCE node
I20260811 11:52:02.562770 22779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:52:02.563735 22779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:52:02.564939 22779 hybrid_clock.cc:648] HybridClock initialized: now 1786449122564910 us; error 46 us; skew 500 ppm
Aug 11 11:52:02 dist-test-slave-bndk krb5kdc[21667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449122, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:52:02.569041 22779 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:52:02.570389 22779 webserver.cc:460] Webserver started at http://127.3.221.66:38763/ using document root <none> and password file <none>
I20260811 11:52:02.571164 22779 fs_manager.cc:359] Metadata directory not provided
I20260811 11:52:02.571273 22779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:52:02.571537 22779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 11:52:02 dist-test-slave-bndk krb5kdc[21667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449122, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
I20260811 11:52:02.602257 22779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "dbea8af6433e4f03ba710a5f4730a21c"
format_stamp: "Formatted at 2026-08-11 11:52:02 on dist-test-slave-bndk"
server_key: "15d8988753593bad3df27a4db2bbbcc2"
server_key_iv: "76e56d312aa2888596fb0c3534dd5189"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:02.603272 22779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "dbea8af6433e4f03ba710a5f4730a21c"
format_stamp: "Formatted at 2026-08-11 11:52:02 on dist-test-slave-bndk"
server_key: "15d8988753593bad3df27a4db2bbbcc2"
server_key_iv: "76e56d312aa2888596fb0c3534dd5189"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:02.610142 22779 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.003s
I20260811 11:52:02.629695 22808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:02.631239 22779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 11:52:02.631394 22779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "dbea8af6433e4f03ba710a5f4730a21c"
format_stamp: "Formatted at 2026-08-11 11:52:02 on dist-test-slave-bndk"
server_key: "15d8988753593bad3df27a4db2bbbcc2"
server_key_iv: "76e56d312aa2888596fb0c3534dd5189"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:02.631534 22779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:52:02.646775 22779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:52:02.650389 22779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:52:02.650650 22779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:52:02.651523 22779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:52:02.652746 22779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:52:02.652818 22779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:02.652879 22779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:52:02.652911 22779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:02.664763 22779 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:43669
I20260811 11:52:02.664876 22978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:43669 every 8 connection(s)
I20260811 11:52:02.666033 22779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:52:02.671723 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 22779
I20260811 11:52:02.671861 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:52:02.672204 3957 external_mini_cluster.cc:1428] Setting key 3ff2b2ad7973118717d85067989196e8
WARNING: no policy specified for kudu/127.3.221.67@KRBTEST.COM; defaulting to no policy
Aug 11 11:52:02 dist-test-slave-bndk krb5kdc[21667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449122, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.67@KRBTEST.COM" created.
I20260811 11:52:02.690598 22774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:02.672795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:40753 (local address 127.3.221.126:38377)
0811 11:52:02.672945 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:02.672950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:02.673580 (+ 630us) server_negotiation.cc:407] Connection header received
0811 11:52:02.677036 (+ 3456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:02.677040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:02.677109 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:02.677166 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:52:02.679919 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:02.680746 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:02.682832 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:02.683195 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:02.683288 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:02.686522 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:52:02.686526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:52:02.686528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:52:02.686555 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:52:02.688021 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:02.688577 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:02.688581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:02.688582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:02.688668 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:02.689114 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:02.689117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:02.689119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:02.689398 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:52:02.689488 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:02.690271 (+ 783us) server_negotiation.cc:299] Negotiation successful
0811 11:52:02.690425 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 11:52:02.692328 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:02.671061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:38377 (local address 127.3.221.66:40753)
0811 11:52:02.673326 (+ 2265us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:02.673396 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:02.675194 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:02.677697 (+ 2503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:52:02.677709 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:52:02.678130 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:52:02.678843 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:02.678863 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:02.680945 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:02.680949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:02.682656 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:02.682671 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:02.683552 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:02.683555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:02.683784 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:02.684561 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:52:02.684584 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:52:02.686305 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:52:02.688164 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:02.688169 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:02.688184 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:02.688456 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:02.688760 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:02.688763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:02.688765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:02.688985 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:02.689570 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:52:02.689577 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:52:02.689953 (+ 376us) client_negotiation.cc:769] Sending connection context
0811 11:52:02.690780 (+ 827us) client_negotiation.cc:240] Negotiation successful
0811 11:52:02.691171 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1990,"thread_start_us":153,"threads_started":1}
I20260811 11:52:02.693905 22979 heartbeater.cc:347] Connected to a master server at 127.3.221.126:38377
I20260811 11:52:02.694181 22979 heartbeater.cc:464] Registering TS with master...
I20260811 11:52:02.694746 22979 heartbeater.cc:511] Master 127.3.221.126:38377 requested a full tablet report, sending...
I20260811 11:52:02.695858 22439 ts_manager.cc:194] Registered new tserver with Master: dbea8af6433e4f03ba710a5f4730a21c (127.3.221.66:43669)
I20260811 11:52:02.696576 22439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:40753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
I20260811 11:52:02.712639 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.221.84:48403/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:38377
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 11:52:02 dist-test-slave-bndk krb5kdc[21667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449122, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
W20260811 11:52:02.871445 22987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:52:02.872452 22987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:52:02.872591 22987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:52:02.882136 22987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:52:02.882309 22987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:52:02.882478 22987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:52:02.894706 22987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.221.84:48403/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:38377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:52:02.895967 22987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:52:02.897089 22987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:52:02.908851 22996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:52:02.909153 22987 server_base.cc:1034] running on GCE node
W20260811 11:52:02.908852 22993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:52:02.908844 22994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:52:02.909899 22987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:52:02.910713 22987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:52:02.911892 22987 hybrid_clock.cc:648] HybridClock initialized: now 1786449122911867 us; error 39 us; skew 500 ppm
Aug 11 11:52:02 dist-test-slave-bndk krb5kdc[21667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449122, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:52:02.914856 22987 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:52:02.915802 22987 webserver.cc:460] Webserver started at http://127.3.221.67:46023/ using document root <none> and password file <none>
I20260811 11:52:02.916391 22987 fs_manager.cc:359] Metadata directory not provided
I20260811 11:52:02.916473 22987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:52:02.916712 22987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 11:52:02 dist-test-slave-bndk krb5kdc[21667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449122, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
I20260811 11:52:02.945735 22987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "13ad49c99d7e4884b94e36bdd0044ae8"
format_stamp: "Formatted at 2026-08-11 11:52:02 on dist-test-slave-bndk"
server_key: "a24e61c2a9b95e55c456e8e20fc15972"
server_key_iv: "bf8d54c17892556b494253eea5705aec"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:02.946756 22987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "13ad49c99d7e4884b94e36bdd0044ae8"
format_stamp: "Formatted at 2026-08-11 11:52:02 on dist-test-slave-bndk"
server_key: "a24e61c2a9b95e55c456e8e20fc15972"
server_key_iv: "bf8d54c17892556b494253eea5705aec"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:02.952987 22987 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260811 11:52:02.971683 23010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:02.973651 22987 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 11:52:02.973898 22987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "13ad49c99d7e4884b94e36bdd0044ae8"
format_stamp: "Formatted at 2026-08-11 11:52:02 on dist-test-slave-bndk"
server_key: "a24e61c2a9b95e55c456e8e20fc15972"
server_key_iv: "bf8d54c17892556b494253eea5705aec"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:02.974157 22987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:52:02.988814 22987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:52:02.992771 22987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:52:02.992991 22987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:52:02.993808 22987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:52:02.994923 22987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:52:02.994984 22987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:02.995035 22987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:52:02.995106 22987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260811 11:52:03.007995 22987 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:45311
I20260811 11:52:03.008054 23180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:45311 every 8 connection(s)
I20260811 11:52:03.009336 22987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:52:03.009960 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 22987
I20260811 11:52:03.010099 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:52:03.010454 3957 external_mini_cluster.cc:1428] Setting key 88644be88393747fee7cc2c825eb7358
Aug 11 11:52:03 dist-test-slave-bndk krb5kdc[21667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449122, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:52:03.031919 22774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:03.017692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:35005 (local address 127.3.221.126:38377)
0811 11:52:03.017817 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:03.017822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:03.017840 (+ 18us) server_negotiation.cc:407] Connection header received
0811 11:52:03.018531 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:03.018535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:03.018598 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:03.018703 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:52:03.020293 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.021154 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.022953 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.023355 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.023463 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:03.027052 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:52:03.027056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:52:03.027059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:52:03.027093 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:52:03.028879 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:03.029768 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:03.029772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:03.029774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:03.029887 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:03.030407 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:03.030411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:03.030413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:03.030690 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:52:03.030805 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:03.031583 (+ 778us) server_negotiation.cc:299] Negotiation successful
0811 11:52:03.031741 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 11:52:03.032971 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:03.016286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:38377 (local address 127.3.221.67:35005)
0811 11:52:03.017003 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:03.017071 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:03.018240 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:03.018926 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:52:03.018936 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:52:03.019373 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:52:03.020102 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:03.020125 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.021345 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.021349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:03.022799 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:03.022813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.023750 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.023755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:03.024035 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:03.024932 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:52:03.024954 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:52:03.026819 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:52:03.029030 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:03.029038 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:03.029059 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:03.029637 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:03.030000 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:03.030004 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:03.030006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:03.030270 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:03.030816 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:52:03.030825 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:52:03.031202 (+ 377us) client_negotiation.cc:769] Sending connection context
0811 11:52:03.031454 (+ 252us) client_negotiation.cc:240] Negotiation successful
0811 11:52:03.031836 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":192,"threads_started":1}
I20260811 11:52:03.034431 23181 heartbeater.cc:347] Connected to a master server at 127.3.221.126:38377
I20260811 11:52:03.034742 23181 heartbeater.cc:464] Registering TS with master...
I20260811 11:52:03.035492 23181 heartbeater.cc:511] Master 127.3.221.126:38377 requested a full tablet report, sending...
I20260811 11:52:03.037153 22439 ts_manager.cc:194] Registered new tserver with Master: 13ad49c99d7e4884b94e36bdd0044ae8 (127.3.221.67:45311)
I20260811 11:52:03.037995 22439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:35005
I20260811 11:52:03.049201 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:52:03.062060 22774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:03.051732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53870 (local address 127.3.221.126:38377)
0811 11:52:03.051874 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:03.051878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:03.052309 (+ 431us) server_negotiation.cc:407] Connection header received
0811 11:52:03.052510 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:03.052514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:03.052576 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:03.052665 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:52:03.053754 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.054576 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.055814 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.056168 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.056299 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:03.058104 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:52:03.058109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:52:03.058112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:52:03.058174 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:52:03.059886 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:03.060509 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:03.060513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:03.060515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:03.060624 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:03.060911 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:03.060914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:03.060916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:03.061174 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:52:03.061372 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:03.061720 (+ 348us) server_negotiation.cc:299] Negotiation successful
0811 11:52:03.061864 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 11:52:03.066080 22439 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:52:03.070904 22439 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:52:03.092926 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:03.085008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:43669 (local address 127.0.0.1:60478)
0811 11:52:03.086590 (+ 1582us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:03.086603 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:03.086687 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:03.087651 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:52:03.087657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:52:03.087681 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:52:03.087828 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:03.087835 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.089439 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.089442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:03.090937 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:03.090950 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.092343 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.092347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:03.092476 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:03.092518 (+ 42us) client_negotiation.cc:769] Sending connection context
0811 11:52:03.092627 (+ 109us) client_negotiation.cc:240] Negotiation successful
0811 11:52:03.092763 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1521,"thread_start_us":462,"threads_started":1}
I20260811 11:52:03.094445 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:03.085797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60478 (local address 127.3.221.66:43669)
0811 11:52:03.086938 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:03.086950 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:03.086975 (+ 25us) server_negotiation.cc:407] Connection header received
0811 11:52:03.087056 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:03.087061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:03.087275 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:03.087469 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:52:03.087968 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.089286 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.091119 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.091992 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.092098 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:03.092274 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:03.094142 (+ 1868us) server_negotiation.cc:299] Negotiation successful
0811 11:52:03.094281 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1031,"thread_start_us":201,"threads_started":1}
I20260811 11:52:03.096304 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:03.084744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:44655 (local address 127.0.0.1:49382)
0811 11:52:03.086034 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:03.086095 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:03.086316 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:03.089335 (+ 3019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:52:03.089339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:52:03.089362 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:52:03.089501 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:03.089510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.092025 (+ 2515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.092029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:03.093570 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:03.093584 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.095864 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.095869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:03.095987 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:03.096019 (+ 32us) client_negotiation.cc:769] Sending connection context
0811 11:52:03.096073 (+ 54us) client_negotiation.cc:240] Negotiation successful
0811 11:52:03.096153 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1092,"thread_start_us":158,"threads_started":1}
I20260811 11:52:03.097359 23202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:03.086279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49382 (local address 127.3.221.65:44655)
0811 11:52:03.088763 (+ 2484us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:03.088770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:03.088796 (+ 26us) server_negotiation.cc:407] Connection header received
0811 11:52:03.088876 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:03.088882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:03.089097 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:03.089671 (+ 574us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:52:03.089806 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.091882 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.093737 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.095697 (+ 1960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.096823 (+ 1126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:03.096999 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:03.097062 (+ 63us) server_negotiation.cc:299] Negotiation successful
0811 11:52:03.097172 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2375,"thread_start_us":106,"threads_started":1}
I20260811 11:52:03.098199 22878 tablet_service.cc:1467] Processing CreateTablet for tablet bdb2a3bf67254c3daefac64e3f695c61 (DEFAULT_TABLE table=test-table [id=21edbcc385a440f190651f8dba356eb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:52:03.100503 22878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb2a3bf67254c3daefac64e3f695c61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:52:03.101114 22665 tablet_service.cc:1467] Processing CreateTablet for tablet bdb2a3bf67254c3daefac64e3f695c61 (DEFAULT_TABLE table=test-table [id=21edbcc385a440f190651f8dba356eb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:52:03.103739 22665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb2a3bf67254c3daefac64e3f695c61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:52:03.115584 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:03.087403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50066 (local address 127.3.221.67:45311)
0811 11:52:03.101595 (+ 14192us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:03.101608 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:03.101641 (+ 33us) server_negotiation.cc:407] Connection header received
0811 11:52:03.101745 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:03.101751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:03.101971 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:03.102261 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:52:03.110936 (+ 8675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.112119 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.113742 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.114628 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.114723 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:03.114898 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:03.115294 (+ 396us) server_negotiation.cc:299] Negotiation successful
0811 11:52:03.115414 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14053,"thread_start_us":203,"threads_started":1}
I20260811 11:52:03.116098 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:03.086909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:45311 (local address 127.0.0.1:50066)
0811 11:52:03.093927 (+ 7018us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:03.093942 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:03.094014 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:03.110609 (+ 16595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:52:03.110613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:52:03.110645 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:52:03.110768 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:03.110775 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.112315 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.112319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:03.113520 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:03.113537 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.115015 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.115019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:03.115119 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:03.115145 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 11:52:03.115860 (+ 715us) client_negotiation.cc:240] Negotiation successful
0811 11:52:03.115961 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6956,"thread_start_us":149,"threads_started":1}
I20260811 11:52:03.120674 23208 tablet_bootstrap.cc:492] T bdb2a3bf67254c3daefac64e3f695c61 P dbea8af6433e4f03ba710a5f4730a21c: Bootstrap starting.
I20260811 11:52:03.120118 23061 tablet_service.cc:1467] Processing CreateTablet for tablet bdb2a3bf67254c3daefac64e3f695c61 (DEFAULT_TABLE table=test-table [id=21edbcc385a440f190651f8dba356eb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:52:03.122906 23061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb2a3bf67254c3daefac64e3f695c61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:52:03.123600 23207 tablet_bootstrap.cc:492] T bdb2a3bf67254c3daefac64e3f695c61 P 89c7a40cb720459ca96f1288a80a03a3: Bootstrap starting.
I20260811 11:52:03.126526 23207 tablet_bootstrap.cc:654] T bdb2a3bf67254c3daefac64e3f695c61 P 89c7a40cb720459ca96f1288a80a03a3: Neither blocks nor log segments found. Creating new log.
I20260811 11:52:03.127568 23207 log.cc:824] T bdb2a3bf67254c3daefac64e3f695c61 P 89c7a40cb720459ca96f1288a80a03a3: Log is configured to *not* fsync() on all Append() calls
I20260811 11:52:03.128996 23208 tablet_bootstrap.cc:654] T bdb2a3bf67254c3daefac64e3f695c61 P dbea8af6433e4f03ba710a5f4730a21c: Neither blocks nor log segments found. Creating new log.
I20260811 11:52:03.130185 23208 log.cc:824] T bdb2a3bf67254c3daefac64e3f695c61 P dbea8af6433e4f03ba710a5f4730a21c: Log is configured to *not* fsync() on all Append() calls
I20260811 11:52:03.132975 23208 tablet_bootstrap.cc:492] T bdb2a3bf67254c3daefac64e3f695c61 P dbea8af6433e4f03ba710a5f4730a21c: No bootstrap required, opened a new log
I20260811 11:52:03.133261 23208 ts_tablet_manager.cc:1397] T bdb2a3bf67254c3daefac64e3f695c61 P dbea8af6433e4f03ba710a5f4730a21c: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260811 11:52:03.137544 23207 tablet_bootstrap.cc:492] T bdb2a3bf67254c3daefac64e3f695c61 P 89c7a40cb720459ca96f1288a80a03a3: No bootstrap required, opened a new log
I20260811 11:52:03.137943 23207 ts_tablet_manager.cc:1397] T bdb2a3bf67254c3daefac64e3f695c61 P 89c7a40cb720459ca96f1288a80a03a3: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20260811 11:52:03.138480 23208 raft_consensus.cc:348] T bdb2a3bf67254c3daefac64e3f695c61 P dbea8af6433e4f03ba710a5f4730a21c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c7a40cb720459ca96f1288a80a03a3" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 44655 } } peers { permanent_uuid: "13ad49c99d7e4884b94e36bdd0044ae8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 45311 } } peers { permanent_uuid: "dbea8af6433e4f03ba710a5f4730a21c" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 43669 } }
I20260811 11:52:03.139271 23208 raft_consensus.cc:374] T bdb2a3bf67254c3daefac64e3f695c61 P dbea8af6433e4f03ba710a5f4730a21c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:52:03.139417 23208 raft_consensus.cc:729] T bdb2a3bf67254c3daefac64e3f695c61 P dbea8af6433e4f03ba710a5f4730a21c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbea8af6433e4f03ba710a5f4730a21c, State: Initialized, Role: FOLLOWER
I20260811 11:52:03.140080 23208 consensus_queue.cc:260] T bdb2a3bf67254c3daefac64e3f695c61 P dbea8af6433e4f03ba710a5f4730a21c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c7a40cb720459ca96f1288a80a03a3" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 44655 } } peers { permanent_uuid: "13ad49c99d7e4884b94e36bdd0044ae8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 45311 } } peers { permanent_uuid: "dbea8af6433e4f03ba710a5f4730a21c" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 43669 } }
I20260811 11:52:03.141556 23208 ts_tablet_manager.cc:1428] T bdb2a3bf67254c3daefac64e3f695c61 P dbea8af6433e4f03ba710a5f4730a21c: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260811 11:52:03.142066 22979 heartbeater.cc:503] Master 127.3.221.126:38377 was elected leader, sending a full tablet report...
I20260811 11:52:03.144230 23212 tablet_bootstrap.cc:492] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8: Bootstrap starting.
I20260811 11:52:03.147661 23212 tablet_bootstrap.cc:654] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8: Neither blocks nor log segments found. Creating new log.
I20260811 11:52:03.148949 23212 log.cc:824] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8: Log is configured to *not* fsync() on all Append() calls
I20260811 11:52:03.151865 23212 tablet_bootstrap.cc:492] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8: No bootstrap required, opened a new log
I20260811 11:52:03.152100 23212 ts_tablet_manager.cc:1397] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260811 11:52:03.156863 23212 raft_consensus.cc:348] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c7a40cb720459ca96f1288a80a03a3" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 44655 } } peers { permanent_uuid: "13ad49c99d7e4884b94e36bdd0044ae8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 45311 } } peers { permanent_uuid: "dbea8af6433e4f03ba710a5f4730a21c" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 43669 } }
I20260811 11:52:03.157334 23212 raft_consensus.cc:374] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:52:03.157140 23207 raft_consensus.cc:348] T bdb2a3bf67254c3daefac64e3f695c61 P 89c7a40cb720459ca96f1288a80a03a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c7a40cb720459ca96f1288a80a03a3" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 44655 } } peers { permanent_uuid: "13ad49c99d7e4884b94e36bdd0044ae8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 45311 } } peers { permanent_uuid: "dbea8af6433e4f03ba710a5f4730a21c" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 43669 } }
I20260811 11:52:03.157480 23207 raft_consensus.cc:374] T bdb2a3bf67254c3daefac64e3f695c61 P 89c7a40cb720459ca96f1288a80a03a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:52:03.157543 23207 raft_consensus.cc:729] T bdb2a3bf67254c3daefac64e3f695c61 P 89c7a40cb720459ca96f1288a80a03a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c7a40cb720459ca96f1288a80a03a3, State: Initialized, Role: FOLLOWER
I20260811 11:52:03.157446 23212 raft_consensus.cc:729] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13ad49c99d7e4884b94e36bdd0044ae8, State: Initialized, Role: FOLLOWER
I20260811 11:52:03.158339 23207 consensus_queue.cc:260] T bdb2a3bf67254c3daefac64e3f695c61 P 89c7a40cb720459ca96f1288a80a03a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c7a40cb720459ca96f1288a80a03a3" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 44655 } } peers { permanent_uuid: "13ad49c99d7e4884b94e36bdd0044ae8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 45311 } } peers { permanent_uuid: "dbea8af6433e4f03ba710a5f4730a21c" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 43669 } }
I20260811 11:52:03.158278 23212 consensus_queue.cc:260] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c7a40cb720459ca96f1288a80a03a3" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 44655 } } peers { permanent_uuid: "13ad49c99d7e4884b94e36bdd0044ae8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 45311 } } peers { permanent_uuid: "dbea8af6433e4f03ba710a5f4730a21c" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 43669 } }
I20260811 11:52:03.160054 23212 ts_tablet_manager.cc:1428] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8: Time spent starting tablet: real 0.008s user 0.002s sys 0.005s
I20260811 11:52:03.160082 23207 ts_tablet_manager.cc:1428] T bdb2a3bf67254c3daefac64e3f695c61 P 89c7a40cb720459ca96f1288a80a03a3: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20260811 11:52:03.160496 22770 heartbeater.cc:503] Master 127.3.221.126:38377 was elected leader, sending a full tablet report...
I20260811 11:52:03.160502 23181 heartbeater.cc:503] Master 127.3.221.126:38377 was elected leader, sending a full tablet report...
W20260811 11:52:03.169801 22983 tablet.cc:2369] T bdb2a3bf67254c3daefac64e3f695c61 P dbea8af6433e4f03ba710a5f4730a21c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 11 11:52:03 dist-test-slave-bndk krb5kdc[21667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449122, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
I20260811 11:52:03.207374 23219 raft_consensus.cc:482] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:52:03.207649 23219 raft_consensus.cc:504] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c7a40cb720459ca96f1288a80a03a3" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 44655 } } peers { permanent_uuid: "13ad49c99d7e4884b94e36bdd0044ae8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 45311 } } peers { permanent_uuid: "dbea8af6433e4f03ba710a5f4730a21c" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 43669 } }
I20260811 11:52:03.209182 23219 leader_election.cc:288] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89c7a40cb720459ca96f1288a80a03a3 (127.3.221.65:44655), dbea8af6433e4f03ba710a5f4730a21c (127.3.221.66:43669)
I20260811 11:52:03.218214 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:03.209437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:44655 (local address 127.3.221.67:34415)
0811 11:52:03.209920 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:03.209936 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:03.210046 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:03.210611 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:52:03.210614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:52:03.210644 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:52:03.210795 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:03.210803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.211941 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.211945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:03.216526 (+ 4581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:03.216543 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.217727 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.217731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:03.217873 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:03.217900 (+ 27us) client_negotiation.cc:769] Sending connection context
0811 11:52:03.217975 (+ 75us) client_negotiation.cc:240] Negotiation successful
0811 11:52:03.218058 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385}
I20260811 11:52:03.218930 23202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:03.210219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:34415 (local address 127.3.221.65:44655)
0811 11:52:03.210357 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:03.210362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:03.210376 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:52:03.210431 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:03.210434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:03.210482 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:03.210559 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:52:03.210957 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.211786 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.216721 (+ 4935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.217470 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.217599 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:03.217655 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:03.218696 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0811 11:52:03.218790 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 11:52:03.220005 22693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb2a3bf67254c3daefac64e3f695c61" candidate_uuid: "13ad49c99d7e4884b94e36bdd0044ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89c7a40cb720459ca96f1288a80a03a3" is_pre_election: true
I20260811 11:52:03.220383 22693 raft_consensus.cc:2454] T bdb2a3bf67254c3daefac64e3f695c61 P 89c7a40cb720459ca96f1288a80a03a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13ad49c99d7e4884b94e36bdd0044ae8 in term 0.
I20260811 11:52:03.221029 23014 leader_election.cc:302] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13ad49c99d7e4884b94e36bdd0044ae8, 89c7a40cb720459ca96f1288a80a03a3; no voters:
I20260811 11:52:03.221493 23219 raft_consensus.cc:2781] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:52:03.221590 23219 raft_consensus.cc:482] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:52:03.221637 23219 raft_consensus.cc:3037] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:52:03.223161 23219 raft_consensus.cc:504] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c7a40cb720459ca96f1288a80a03a3" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 44655 } } peers { permanent_uuid: "13ad49c99d7e4884b94e36bdd0044ae8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 45311 } } peers { permanent_uuid: "dbea8af6433e4f03ba710a5f4730a21c" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 43669 } }
I20260811 11:52:03.223608 23219 leader_election.cc:288] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8 [CANDIDATE]: Term 1 election: Requested vote from peers 89c7a40cb720459ca96f1288a80a03a3 (127.3.221.65:44655), dbea8af6433e4f03ba710a5f4730a21c (127.3.221.66:43669)
I20260811 11:52:03.224815 22693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb2a3bf67254c3daefac64e3f695c61" candidate_uuid: "13ad49c99d7e4884b94e36bdd0044ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89c7a40cb720459ca96f1288a80a03a3"
I20260811 11:52:03.224975 22693 raft_consensus.cc:3037] T bdb2a3bf67254c3daefac64e3f695c61 P 89c7a40cb720459ca96f1288a80a03a3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:52:03.226531 22693 raft_consensus.cc:2454] T bdb2a3bf67254c3daefac64e3f695c61 P 89c7a40cb720459ca96f1288a80a03a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13ad49c99d7e4884b94e36bdd0044ae8 in term 1.
I20260811 11:52:03.227252 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:03.209539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:43669 (local address 127.3.221.67:42411)
0811 11:52:03.216255 (+ 6716us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:03.216274 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:03.216505 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:03.218277 (+ 1772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:52:03.218280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:52:03.218303 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:52:03.218444 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:03.218453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.224283 (+ 5830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.224287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:03.225599 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:03.225614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.226735 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.226739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:03.226843 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:03.226868 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 11:52:03.226937 (+ 69us) client_negotiation.cc:240] Negotiation successful
0811 11:52:03.227034 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6603,"thread_start_us":156,"threads_started":1}
I20260811 11:52:03.227262 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:03.210018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:42411 (local address 127.3.221.66:43669)
0811 11:52:03.210142 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:03.210147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:03.216387 (+ 6240us) server_negotiation.cc:407] Connection header received
0811 11:52:03.216678 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:03.216682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:03.216750 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:03.216805 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:52:03.218602 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.219315 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.225781 (+ 6466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.226429 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.226546 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:03.226604 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:03.227025 (+ 421us) server_negotiation.cc:299] Negotiation successful
0811 11:52:03.227133 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 11:52:03.229656 22911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb2a3bf67254c3daefac64e3f695c61" candidate_uuid: "13ad49c99d7e4884b94e36bdd0044ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbea8af6433e4f03ba710a5f4730a21c"
I20260811 11:52:03.229955 22911 raft_consensus.cc:3037] T bdb2a3bf67254c3daefac64e3f695c61 P dbea8af6433e4f03ba710a5f4730a21c [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:52:03.231482 22911 raft_consensus.cc:2454] T bdb2a3bf67254c3daefac64e3f695c61 P dbea8af6433e4f03ba710a5f4730a21c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13ad49c99d7e4884b94e36bdd0044ae8 in term 1.
I20260811 11:52:03.231948 22894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb2a3bf67254c3daefac64e3f695c61" candidate_uuid: "13ad49c99d7e4884b94e36bdd0044ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbea8af6433e4f03ba710a5f4730a21c" is_pre_election: true
I20260811 11:52:03.232053 23023 leader_election.cc:302] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13ad49c99d7e4884b94e36bdd0044ae8, dbea8af6433e4f03ba710a5f4730a21c; no voters:
I20260811 11:52:03.232097 22894 raft_consensus.cc:2362] T bdb2a3bf67254c3daefac64e3f695c61 P dbea8af6433e4f03ba710a5f4730a21c [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 13ad49c99d7e4884b94e36bdd0044ae8 in term 1. Re-sending same reply.
I20260811 11:52:03.233366 23219 raft_consensus.cc:2781] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:52:03.233644 23219 raft_consensus.cc:686] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8 [term 1 LEADER]: Becoming Leader. State: Replica: 13ad49c99d7e4884b94e36bdd0044ae8, State: Running, Role: LEADER
I20260811 11:52:03.234032 23219 consensus_queue.cc:237] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c7a40cb720459ca96f1288a80a03a3" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 44655 } } peers { permanent_uuid: "13ad49c99d7e4884b94e36bdd0044ae8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 45311 } } peers { permanent_uuid: "dbea8af6433e4f03ba710a5f4730a21c" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 43669 } }
I20260811 11:52:03.237880 22432 catalog_manager.cc:5495] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13ad49c99d7e4884b94e36bdd0044ae8 (127.3.221.67). New cstate: current_term: 1 leader_uuid: "13ad49c99d7e4884b94e36bdd0044ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c7a40cb720459ca96f1288a80a03a3" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 44655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13ad49c99d7e4884b94e36bdd0044ae8" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 45311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbea8af6433e4f03ba710a5f4730a21c" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 43669 } health_report { overall_health: UNKNOWN } } }
I20260811 11:52:03.258886 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:03.253940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50076 (local address 127.3.221.67:45311)
0811 11:52:03.254062 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:03.254067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:03.254252 (+ 185us) server_negotiation.cc:407] Connection header received
0811 11:52:03.254357 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:03.254361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:03.254453 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:03.254567 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:52:03.254949 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.255457 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.256326 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:03.256555 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:03.256644 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:03.257512 (+ 868us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:52:03.258140 (+ 628us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:52:03.258274 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:03.258657 (+ 383us) server_negotiation.cc:299] Negotiation successful
0811 11:52:03.258731 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260811 11:52:03.263984 23182 tablet.cc:2369] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:52:03.266621 22894 raft_consensus.cc:1261] T bdb2a3bf67254c3daefac64e3f695c61 P dbea8af6433e4f03ba710a5f4730a21c [term 1 FOLLOWER]: Refusing update from remote peer 13ad49c99d7e4884b94e36bdd0044ae8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:52:03.266620 22693 raft_consensus.cc:1261] T bdb2a3bf67254c3daefac64e3f695c61 P 89c7a40cb720459ca96f1288a80a03a3 [term 1 FOLLOWER]: Refusing update from remote peer 13ad49c99d7e4884b94e36bdd0044ae8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:52:03.267472 23225 consensus_queue.cc:1035] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89c7a40cb720459ca96f1288a80a03a3" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 44655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:52:03.268010 23225 consensus_queue.cc:1035] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbea8af6433e4f03ba710a5f4730a21c" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 43669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:52:03.278239 23230 mvcc.cc:205] Tried to move back new op lower bound from 7317295608890765312 to 7317295608770478080. Current Snapshot: MvccSnapshot[applied={T|T < 7317295608890765312}]
I20260811 11:52:03.281639 23232 mvcc.cc:205] Tried to move back new op lower bound from 7317295608890765312 to 7317295608770478080. Current Snapshot: MvccSnapshot[applied={T|T < 7317295608890765312}]
I20260811 11:52:03.285662 23234 mvcc.cc:205] Tried to move back new op lower bound from 7317295608890765312 to 7317295608770478080. Current Snapshot: MvccSnapshot[applied={T|T < 7317295608890765312}]
I20260811 11:52:03.296702 22439 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:52:03.296926 22439 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:52:03.300364 22439 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7cff43c5ee314d4385188acc81124764: Sending DeleteTablet for 3 replicas of tablet bdb2a3bf67254c3daefac64e3f695c61
I20260811 11:52:03.301841 22665 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb2a3bf67254c3daefac64e3f695c61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:52:03 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:49382
I20260811 11:52:03.301973 22878 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb2a3bf67254c3daefac64e3f695c61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:52:03 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:60478
I20260811 11:52:03.303301 23061 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb2a3bf67254c3daefac64e3f695c61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:52:03 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:50066
I20260811 11:52:03.304245 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 22561
I20260811 11:52:03.336316 23248 tablet_replica.cc:330] T bdb2a3bf67254c3daefac64e3f695c61 P dbea8af6433e4f03ba710a5f4730a21c: stopping tablet replica
I20260811 11:52:03.341694 23248 raft_consensus.cc:2229] T bdb2a3bf67254c3daefac64e3f695c61 P dbea8af6433e4f03ba710a5f4730a21c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:52:03.342322 23248 raft_consensus.cc:2258] T bdb2a3bf67254c3daefac64e3f695c61 P dbea8af6433e4f03ba710a5f4730a21c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:52:03.344419 23248 ts_tablet_manager.cc:1905] T bdb2a3bf67254c3daefac64e3f695c61 P dbea8af6433e4f03ba710a5f4730a21c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:52:03.345618 23249 tablet_replica.cc:330] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8: stopping tablet replica
I20260811 11:52:03.346072 23249 raft_consensus.cc:2229] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:52:03.353375 23249 raft_consensus.cc:2258] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:52:03.356854 23248 ts_tablet_manager.cc:1918] T bdb2a3bf67254c3daefac64e3f695c61 P dbea8af6433e4f03ba710a5f4730a21c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:52:03.359118 23249 ts_tablet_manager.cc:1905] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260811 11:52:03.360590 22412 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.221.65:44655 (error 108)
I20260811 11:52:03.360910 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 22779
I20260811 11:52:03.365059 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:03.364605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:44655 (local address 127.0.0.1:49390)
0811 11:52:03.364850 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:03.364945 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.65:44655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260811 11:52:03.365329 22412 catalog_manager.cc:4584] TS 89c7a40cb720459ca96f1288a80a03a3 (127.3.221.65:44655): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bdb2a3bf67254c3daefac64e3f695c61: Network error: Client connection negotiation failed: client connection to 127.3.221.65:44655: connect: Connection refused (error 111)
I20260811 11:52:03.391673 23249 ts_tablet_manager.cc:1918] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:52:03.392814 23249 log.cc:1196] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/bdb2a3bf67254c3daefac64e3f695c61
I20260811 11:52:03.393474 23249 ts_tablet_manager.cc:1939] T bdb2a3bf67254c3daefac64e3f695c61 P 13ad49c99d7e4884b94e36bdd0044ae8: Deleting consensus metadata
I20260811 11:52:03.395515 22411 catalog_manager.cc:4841] TS 13ad49c99d7e4884b94e36bdd0044ae8 (127.3.221.67:45311): tablet bdb2a3bf67254c3daefac64e3f695c61 (table test-table [id=21edbcc385a440f190651f8dba356eb2]) successfully deleted
W20260811 11:52:03.409559 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:52:03.408900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:43669 (local address 127.0.0.1:60494)
0811 11:52:03.409099 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:03.409114 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:03.409208 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:03.409429 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.66:43669: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":118}
I20260811 11:52:03.410239 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 22987
I20260811 11:52:03.421109 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:03.420688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:44655 (local address 127.0.0.1:49402)
0811 11:52:03.420928 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:03.420996 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.65:44655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260811 11:52:03.422129 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:03.421848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:44655 (local address 127.0.0.1:49416)
0811 11:52:03.421998 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:03.422049 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.65:44655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260811 11:52:03.448402 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 22381
I20260811 11:52:03.467773 3957 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260811 11:52:03.814095 3957 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260811 11:52:03.814249 3957 mini_ranger.cc:66] Stopping Ranger...
I20260811 11:52:04.160063 3957 mini_ranger.cc:68] Stopped Ranger
2026-08-11 11:52:04.160 UTC [21710] LOG: received smart shutdown request
2026-08-11 11:52:04.164 UTC [21710] LOG: background worker "logical replication launcher" (PID 21723) exited with exit code 1
2026-08-11 11:52:04.165 UTC [21716] LOG: shutting down
2026-08-11 11:52:04.350 UTC [21710] LOG: database system is shut down
2026-08-11T11:52:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50035 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:52:04 dist-test-slave-bndk krb5kdc[23266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:52:04 dist-test-slave-bndk krb5kdc[23266](info): set up 2 sockets
Aug 11 11:52:04 dist-test-slave-bndk krb5kdc[23266](info): commencing operation
krb5kdc: starting...
W20260811 11:52:05.621840 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:52:05 dist-test-slave-bndk krb5kdc[23266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:52:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:52:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/rangeradmin_127.3.221.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/rangeradmin_127.3.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/rangerlookup_127.3.221.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/rangerlookup_127.3.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/HTTP_127.3.221.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/HTTP_127.3.221.84@KRBTEST.COM.keytab.
I20260811 11:52:05.810200 3957 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskt5cui_/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/postgres -l logfile start
2026-08-11 11:52:08.371 UTC [23310] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-11 11:52:08.372 UTC [23310] LOG: listening on IPv4 address "127.3.221.84", port 35317
2026-08-11 11:52:08.375 UTC [23310] LOG: listening on Unix socket "/tmp/.s.PGSQL.35317"
2026-08-11 11:52:08.397 UTC [23314] LOG: database system was shut down at 2026-08-11 11:52:06 UTC
2026-08-11 11:52:08.402 UTC [23310] LOG: database system is ready to accept connections
I20260811 11:52:09.472024 3957 mini_postgres.cc:96] Postgres bound to 35317
2026-08-11 11:52:09.479 UTC [23329] FATAL: database "slave" does not exist
127.3.221.84:35317 - accepting connections
I20260811 11:52:09.480461 3957 mini_ranger.cc:161] Starting Ranger...
I20260811 11:52:09.497463 3957 mini_ranger.cc:84] Created miniranger Postgres user
I20260811 11:52:10.048898 3957 mini_ranger.cc:87] Created ranger Postgres database
I20260811 11:52:10.048991 3957 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ranger-admin
2026-08-11 11:52:10,332 [I] DB FLAVOR :POSTGRES
2026-08-11 11:52:10,333 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 11:52:10,333 [I] Checking connection..
2026-08-11 11:52:10,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-11 11:52:10,605 [I] Checking connection passed.
2026-08-11 11:52:10,605 [I] --------- Verifying version history table ---------
2026-08-11 11:52:10,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 11:52:10,865 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 11:52:10,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 11:52:11,128 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 11:52:11,129 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-11 11:52:11,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-11 11:52:11.370 UTC [23429] WARNING: there is no transaction in progress
2026-08-11 11:52:11,379 [I] create_dbversion_catalog.sql file imported successfully
2026-08-11 11:52:11,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 11:52:11,648 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-11 11:52:11,648 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 11:52:11,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-11 11:52:11,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-11 11:52:12,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-bndk.c.gcp-upstream.internal','N') ;"
2026-08-11 11:52:12,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 11:52:12,664 [I] Table x_portal_user does not exist in database ranger
2026-08-11 11:52:12,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 11:52:12,937 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-11 11:52:12,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 11:52:13,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 11:52:13,443 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-11 11:52:13,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-11 11:52:14.518 UTC [23587] WARNING: there is no transaction in progress
2026-08-11 11:52:14.535 UTC [23587] WARNING: there is no transaction in progress
2026-08-11 11:52:14.554 UTC [23587] WARNING: there is no transaction in progress
2026-08-11 11:52:14.571 UTC [23587] WARNING: there is no transaction in progress
2026-08-11 11:52:14.591 UTC [23587] WARNING: there is no transaction in progress
2026-08-11 11:52:14.610 UTC [23587] WARNING: there is no transaction in progress
2026-08-11 11:52:14.677 UTC [23587] WARNING: there is no transaction in progress
2026-08-11 11:52:14.685 UTC [23587] WARNING: there is no transaction in progress
2026-08-11 11:52:14.700 UTC [23587] WARNING: there is no transaction in progress
2026-08-11 11:52:14.710 UTC [23587] WARNING: there is no transaction in progress
2026-08-11 11:52:14.720 UTC [23587] WARNING: there is no transaction in progress
2026-08-11 11:52:14.735 UTC [23587] WARNING: there is no transaction in progress
2026-08-11 11:52:14.745 UTC [23587] WARNING: there is no transaction in progress
2026-08-11 11:52:14.752 UTC [23587] WARNING: there is no transaction in progress
2026-08-11 11:52:14.759 UTC [23587] WARNING: there is no transaction in progress
2026-08-11 11:52:14.768 UTC [23587] WARNING: there is no transaction in progress
2026-08-11 11:52:15,529 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-11 11:52:15,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-11 11:52:15,771 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-11 11:52:15,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 11:52:16,039 [I] Table x_portal_user already exists in database 'ranger'
2026-08-11 11:52:16,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 11:52:16,306 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-11 11:52:16,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 11:52:16,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 11:52:16,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-bndk.c.gcp-upstream.internal';"
2026-08-11 11:52:17,129 [I] CORE_DB_SCHEMA import status has been updated
2026-08-11 11:52:17,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-11 11:52:17,374 [I] DB_PATCHES have already been applied
I20260811 11:52:17.380363 3957 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ranger-admin:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskt5cui_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260811 11:52:17.380439 3957 mini_ranger.cc:193] Using host: 127.3.221.84
I20260811 11:52:17.385180 3957 mini_ranger.cc:227] Ranger admin URL: http://127.3.221.84:54963
Aug 11, 2026 11:52:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 11:52:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 11:52:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ranger-admin/ews/webapp
Aug 11, 2026 11:52:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 11, 2026 11:52:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 11, 2026 11:52:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ranger-admin/ews/webapp].
Aug 11, 2026 11:52:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260811 11:52:18.608387 3957 mini_ranger.cc:160] Time spent starting Ranger: real 9.128s user 0.000s sys 0.013s
Aug 11, 2026 11:52:41 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 11, 2026 11:52:41 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 11, 2026 11:52:41 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 11, 2026 11:52:41 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 11, 2026 11:52:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 11, 2026 11:52:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 11, 2026 11:52:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 11, 2026 11:52:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 11, 2026 11:52:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 11, 2026 11:52:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 11, 2026 11:52:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 11, 2026 11:52:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 11, 2026 11:52:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 11, 2026 11:52:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 11, 2026 11:52:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 11, 2026 11:52:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 11, 2026 11:52:41 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 11, 2026 11:52:42 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260811 11:52:42.853078 3957 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/rangerkms_127.3.221.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/rangerkms_127.3.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.221.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.221.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.221.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/HTTP_127.3.221.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.221.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/HTTP_127.3.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 11 11:52:42 dist-test-slave-bndk krb5kdc[23266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449162, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260811 11:52:42.970598 3957 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260811 11:52:42.984160 3957 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260811 11:52:43.902557 3957 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260811 11:52:43.902653 3957 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ranger-kms
I20260811 11:52:43.902670 3957 mini_ranger_kms.cc:227] Using postgres at 127.3.221.84:35317
2026-08-11 11:52:44,102 [I] DB FLAVOR :POSTGRES
2026-08-11 11:52:44,103 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 11:52:44,103 [I] Checking connection
2026-08-11 11:52:44,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-11 11:52:44,401 [I] connection success
2026-08-11 11:52:44,401 [I] --------- Verifying Ranger DB tables ---------
2026-08-11 11:52:44,401 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-11 11:52:44,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-11 11:52:44,658 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-11 11:52:44,658 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 11:52:44,658 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-11 11:52:44,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.221.84:35317/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-11 11:52:44,953 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260811 11:52:45.518441 3957 mini_ranger_kms.cc:310] Created kms service
I20260811 11:52:45.713408 3957 mini_ranger_kms.cc:326] Created kudu user
I20260811 11:52:45.763906 3957 mini_ranger_kms.cc:343] Created rangerkms user
Aug 11 11:52:45 dist-test-slave-bndk krb5kdc[23266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449162, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
I20260811 11:52:46.241572 3957 mini_ranger_kms.cc:384] Added ranger policy
I20260811 11:52:46.241735 3957 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ranger-kms:/tmp/dist-test-taskt5cui_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskt5cui_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260811 11:52:46.241766 3957 mini_ranger_kms.cc:240] Using host: 127.3.221.84
I20260811 11:52:46.246728 3957 mini_ranger_kms.cc:276] Ranger KMS PID: 23855
I20260811 11:52:46.246835 3957 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.221.84:48725
Aug 11, 2026 11:52:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 11:52:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 11:52:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 11, 2026 11:52:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 11, 2026 11:52:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ranger-kms/ews/webapp].
Aug 11, 2026 11:52:47 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 11 11:52:48 dist-test-slave-bndk krb5kdc[23266](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786449168, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.221.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:52:48.876965 3957 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.906s user 0.016s sys 0.164s
I20260811 11:52:48.877080 3957 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260811 11:52:48.877138 3957 mini_ranger_kms.cc:401] 127.3.221.84:48725/kms/v1/keys
Aug 11 11:52:49 dist-test-slave-bndk krb5kdc[23266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449168, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.221.84@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
Aug 11 11:52:49 dist-test-slave-bndk krb5kdc[23266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449168, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.221.84@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
Aug 11 11:52:51 dist-test-slave-bndk krb5kdc[23266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:52:51.936215 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:42831
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:42831
--ranger_config_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.221.84:48725/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:52:52.073186 23930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:52:52.073513 23930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:52:52.073555 23930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:52:52.077661 23930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:52:52.077732 23930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:52:52.077750 23930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:52:52.077765 23930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:52:52.077780 23930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:52:52.083925 23930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.221.84:48725/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:42831
--ranger_config_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:42831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:52:52.085132 23930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:52:52.086256 23930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:52:52.096093 23939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:52:52.096263 23930 server_base.cc:1034] running on GCE node
W20260811 11:52:52.096103 23941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:52:52.096102 23938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:52:52.097081 23930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:52:52.098397 23930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:52:52.099622 23930 hybrid_clock.cc:648] HybridClock initialized: now 1786449172099600 us; error 37 us; skew 500 ppm
Aug 11 11:52:52 dist-test-slave-bndk krb5kdc[23266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449172, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:52:52.106103 23930 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:52:52.107228 23930 webserver.cc:460] Webserver started at http://127.3.221.126:34839/ using document root <none> and password file <none>
I20260811 11:52:52.107815 23930 fs_manager.cc:359] Metadata directory not provided
I20260811 11:52:52.107880 23930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:52:52.108095 23930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 11:52:52 dist-test-slave-bndk krb5kdc[23266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449172, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
I20260811 11:52:52.166931 23930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "4cc67b9dac514daeb5175042e57f5cc4"
format_stamp: "Formatted at 2026-08-11 11:52:52 on dist-test-slave-bndk"
server_key: "b1ac482b5bd8814dfd0b7b0c7f2020e3"
server_key_iv: "611c2e7242459226cacca2757f794f6c"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:52.167694 23930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "4cc67b9dac514daeb5175042e57f5cc4"
format_stamp: "Formatted at 2026-08-11 11:52:52 on dist-test-slave-bndk"
server_key: "b1ac482b5bd8814dfd0b7b0c7f2020e3"
server_key_iv: "611c2e7242459226cacca2757f794f6c"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:52.172848 23930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 11:52:52.188066 23950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:52.189561 23930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 11:52:52.189721 23930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "4cc67b9dac514daeb5175042e57f5cc4"
format_stamp: "Formatted at 2026-08-11 11:52:52 on dist-test-slave-bndk"
server_key: "b1ac482b5bd8814dfd0b7b0c7f2020e3"
server_key_iv: "611c2e7242459226cacca2757f794f6c"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:52.189854 23930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:52:52.207672 23930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:52:52.210686 23930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:52:52.210889 23930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:52:52.218941 23930 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:42831
I20260811 11:52:52.218951 24035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:42831 every 8 connection(s)
I20260811 11:52:52.220009 23930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:52:52.223047 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:52:52.226974 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 23930
I20260811 11:52:52.227111 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:52:52.227434 3957 external_mini_cluster.cc:1428] Setting key 9b86620171f2ab67d7215126550a0ac9
I20260811 11:52:52.229768 24036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4: Bootstrap starting.
I20260811 11:52:52.232300 24036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4: Neither blocks nor log segments found. Creating new log.
I20260811 11:52:52.233317 24036 log.cc:824] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4: Log is configured to *not* fsync() on all Append() calls
I20260811 11:52:52.235370 24036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4: No bootstrap required, opened a new log
Aug 11 11:52:52 dist-test-slave-bndk krb5kdc[23266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:52:52.239943 24036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc67b9dac514daeb5175042e57f5cc4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42831 } }
I20260811 11:52:52.240212 24036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:52:52.240268 24036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc67b9dac514daeb5175042e57f5cc4, State: Initialized, Role: FOLLOWER
I20260811 11:52:52.240811 24036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc67b9dac514daeb5175042e57f5cc4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42831 } }
I20260811 11:52:52.241014 24036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:52:52.241092 24036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:52:52.241199 24036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:52:52.242476 24036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc67b9dac514daeb5175042e57f5cc4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42831 } }
I20260811 11:52:52.242895 24036 leader_election.cc:302] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cc67b9dac514daeb5175042e57f5cc4; no voters:
I20260811 11:52:52.243270 24036 leader_election.cc:288] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:52:52.243448 24044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:52:52.243693 24044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [term 1 LEADER]: Becoming Leader. State: Replica: 4cc67b9dac514daeb5175042e57f5cc4, State: Running, Role: LEADER
I20260811 11:52:52.244153 24044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc67b9dac514daeb5175042e57f5cc4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42831 } }
I20260811 11:52:52.245337 24036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:52:52.245951 24044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cc67b9dac514daeb5175042e57f5cc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc67b9dac514daeb5175042e57f5cc4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42831 } } }
I20260811 11:52:52.247424 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:52.229877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46626 (local address 127.3.221.126:42831)
0811 11:52:52.230422 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:52.230443 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:52.230493 (+ 50us) server_negotiation.cc:407] Connection header received
0811 11:52:52.231376 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:52.231413 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:52.231898 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:52.232306 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:52:52.233254 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:52.235177 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:52.236395 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:52.237137 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:52.237354 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:52.240682 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:52:52.240689 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:52:52.240708 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:52:52.240745 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:52:52.242768 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:52.243353 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:52.243360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:52.243367 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:52.243515 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:52.243828 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:52.243831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:52.243833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:52.244456 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:52:52.245387 (+ 931us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:52.245544 (+ 157us) server_negotiation.cc:299] Negotiation successful
0811 11:52:52.245843 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":171,"threads_started":1}
I20260811 11:52:52.247618 24036 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260811 11:52:52.246436 24046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cc67b9dac514daeb5175042e57f5cc4. Latest consensus state: current_term: 1 leader_uuid: "4cc67b9dac514daeb5175042e57f5cc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc67b9dac514daeb5175042e57f5cc4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42831 } } }
I20260811 11:52:52.249131 24046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [sys.catalog]: This master's current role is: LEADER
I20260811 11:52:52.246158 24044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [sys.catalog]: This master's current role is: LEADER
I20260811 11:52:52.253206 24068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:52:52.257527 24068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:52:52.265642 24068 catalog_manager.cc:1347] Generated new cluster ID: c115707693d04cd5ba296482a9eabf3a
I20260811 11:52:52.265734 24068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:52:52.304570 24068 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:52:52.305989 24068 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:52:52.313699 24068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4: Generated new TSK 0
I20260811 11:52:52.314661 24068 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
I20260811 11:52:53.118376 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.221.84:48725/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:42831
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:52:53.306143 24101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:52:53.306558 24101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:52:53.306634 24101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:52:53.312994 24101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:52:53.313076 24101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:52:53.313166 24101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:52:53.320530 24101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.221.84:48725/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:52:53.322420 24101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:52:53.324019 24101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:52:53.339520 24111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:52:53.339803 24101 server_base.cc:1034] running on GCE node
W20260811 11:52:53.339550 24112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:52:53.340382 24114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:52:53.341004 24101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:52:53.341939 24101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:52:53.343216 24101 hybrid_clock.cc:648] HybridClock initialized: now 1786449173343137 us; error 98 us; skew 500 ppm
Aug 11 11:52:53 dist-test-slave-bndk krb5kdc[23266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449173, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:52:53.346436 24101 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:52:53.347697 24101 webserver.cc:460] Webserver started at http://127.3.221.65:36889/ using document root <none> and password file <none>
I20260811 11:52:53.348465 24101 fs_manager.cc:359] Metadata directory not provided
I20260811 11:52:53.348532 24101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:52:53.348722 24101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 11:52:53 dist-test-slave-bndk krb5kdc[23266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449173, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
I20260811 11:52:53.373246 24101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "09f190a507c34a28b35fc407add36bfd"
format_stamp: "Formatted at 2026-08-11 11:52:53 on dist-test-slave-bndk"
server_key: "40d758632086ebc80e4c5062fa6dc497"
server_key_iv: "279a0490011eccab140c4f704f07e418"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:53.374169 24101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "09f190a507c34a28b35fc407add36bfd"
format_stamp: "Formatted at 2026-08-11 11:52:53 on dist-test-slave-bndk"
server_key: "40d758632086ebc80e4c5062fa6dc497"
server_key_iv: "279a0490011eccab140c4f704f07e418"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:53.380316 24101 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260811 11:52:53.405753 24125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:53.407620 24101 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260811 11:52:53.407802 24101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "09f190a507c34a28b35fc407add36bfd"
format_stamp: "Formatted at 2026-08-11 11:52:53 on dist-test-slave-bndk"
server_key: "40d758632086ebc80e4c5062fa6dc497"
server_key_iv: "279a0490011eccab140c4f704f07e418"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:53.407964 24101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:52:53.441260 24101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:52:53.445149 24101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:52:53.445624 24101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:52:53.446581 24101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:52:53.447954 24101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:52:53.448069 24101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:53.448277 24101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:52:53.448369 24101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:53.461689 24101 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:33277
I20260811 11:52:53.464371 24297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:33277 every 8 connection(s)
I20260811 11:52:53.465246 24101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:52:53.471532 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 24101
I20260811 11:52:53.471935 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:52:53.472394 3957 external_mini_cluster.cc:1428] Setting key 6afd72490aacc1e224667a48d047eebd
WARNING: no policy specified for kudu/127.3.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.66@KRBTEST.COM" created.
Aug 11 11:52:53 dist-test-slave-bndk krb5kdc[23266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449173, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:52:53.513827 24302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:53.475455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:42403 (local address 127.3.221.126:42831)
0811 11:52:53.484400 (+ 8945us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:53.484406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:53.484423 (+ 17us) server_negotiation.cc:407] Connection header received
0811 11:52:53.484469 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:53.484472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:53.484526 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:53.484624 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:52:53.488375 (+ 3751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:53.489288 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:53.491079 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:53.491439 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:53.491523 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:53.507864 (+ 16341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:52:53.507869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:52:53.507871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:52:53.507912 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:52:53.510013 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:53.511064 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:53.511069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:53.511083 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:53.511199 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:53.511790 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:53.511795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:53.511797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:53.512077 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:52:53.512163 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:53.513251 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0811 11:52:53.513418 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8840,"thread_start_us":110,"threads_started":1}
I20260811 11:52:53.516294 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:53.474133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:42831 (local address 127.3.221.65:42403)
0811 11:52:53.482447 (+ 8314us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:53.482552 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:53.483813 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:53.486920 (+ 3107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:52:53.486932 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:52:53.487333 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:52:53.488148 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:53.488172 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:53.489462 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:53.489466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:53.490914 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:53.490929 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:53.491771 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:53.491776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:53.492017 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:53.503691 (+ 11674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:52:53.503724 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:52:53.507555 (+ 3831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:52:53.510307 (+ 2752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:53.510318 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:53.510343 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:53.510777 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:53.511361 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:53.511364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:53.511366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:53.511662 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:53.512414 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:52:53.512423 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:52:53.512828 (+ 405us) client_negotiation.cc:769] Sending connection context
0811 11:52:53.514528 (+ 1700us) client_negotiation.cc:240] Negotiation successful
0811 11:52:53.515043 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7852,"thread_start_us":158,"threads_started":1}
I20260811 11:52:53.518819 24298 heartbeater.cc:347] Connected to a master server at 127.3.221.126:42831
I20260811 11:52:53.519122 24298 heartbeater.cc:464] Registering TS with master...
I20260811 11:52:53.519847 24298 heartbeater.cc:511] Master 127.3.221.126:42831 requested a full tablet report, sending...
I20260811 11:52:53.522162 23981 ts_manager.cc:194] Registered new tserver with Master: 09f190a507c34a28b35fc407add36bfd (127.3.221.65:33277)
I20260811 11:52:53.523839 23981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:42403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
I20260811 11:52:53.528504 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.221.84:48725/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:42831
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 11:52:53 dist-test-slave-bndk krb5kdc[23266](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786449173, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260811 11:52:53.699537 24305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:52:53.699838 24305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:52:53.699900 24305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:52:53.704232 24305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:52:53.704308 24305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:52:53.704392 24305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:52:53.710623 24305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.221.84:48725/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:52:53.711899 24305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:52:53.713389 24305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:52:53.725265 24316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:52:53.725266 24322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:52:53.725822 24305 server_base.cc:1034] running on GCE node
W20260811 11:52:53.726025 24320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:52:53.726468 24305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:52:53.727313 24305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:52:53.728540 24305 hybrid_clock.cc:648] HybridClock initialized: now 1786449173728507 us; error 38 us; skew 500 ppm
Aug 11 11:52:53 dist-test-slave-bndk krb5kdc[23266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449173, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:52:53.732380 24305 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:52:53.733671 24305 webserver.cc:460] Webserver started at http://127.3.221.66:39437/ using document root <none> and password file <none>
I20260811 11:52:53.734297 24305 fs_manager.cc:359] Metadata directory not provided
I20260811 11:52:53.734377 24305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:52:53.734613 24305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 11:52:53 dist-test-slave-bndk krb5kdc[23266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449173, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
I20260811 11:52:53.759006 24305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "5b9318b4d8314adfa368a3a85228fca5"
format_stamp: "Formatted at 2026-08-11 11:52:53 on dist-test-slave-bndk"
server_key: "febe65b4953a0d4024670993d92d6233"
server_key_iv: "b0431f44654b65182bfd5a14315544f1"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:53.761080 24305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "5b9318b4d8314adfa368a3a85228fca5"
format_stamp: "Formatted at 2026-08-11 11:52:53 on dist-test-slave-bndk"
server_key: "febe65b4953a0d4024670993d92d6233"
server_key_iv: "b0431f44654b65182bfd5a14315544f1"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:53.771152 24305 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.000s sys 0.004s
I20260811 11:52:53.792603 24334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:53.799523 24305 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260811 11:52:53.799942 24305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "5b9318b4d8314adfa368a3a85228fca5"
format_stamp: "Formatted at 2026-08-11 11:52:53 on dist-test-slave-bndk"
server_key: "febe65b4953a0d4024670993d92d6233"
server_key_iv: "b0431f44654b65182bfd5a14315544f1"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:53.800225 24305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:52:53.836532 24305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:52:53.845014 24305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:52:53.845286 24305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:52:53.846163 24305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:52:53.847388 24305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:52:53.847453 24305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:53.847512 24305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:52:53.847543 24305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:53.858842 24305 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:32805
I20260811 11:52:53.859004 24500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:32805 every 8 connection(s)
I20260811 11:52:53.860114 24305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:52:53.861019 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 24305
I20260811 11:52:53.861161 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:52:53.861791 3957 external_mini_cluster.cc:1428] Setting key d4944f9ebf10276a0e4d23b9f3074819
WARNING: no policy specified for kudu/127.3.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.67@KRBTEST.COM" created.
Aug 11 11:52:53 dist-test-slave-bndk krb5kdc[23266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449173, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:52:53.907749 24302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:53.862904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:44115 (local address 127.3.221.126:42831)
0811 11:52:53.863050 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:53.863055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:53.864417 (+ 1362us) server_negotiation.cc:407] Connection header received
0811 11:52:53.866013 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:53.866016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:53.866073 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:53.866127 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:52:53.868066 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:53.868940 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:53.890029 (+ 21089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:53.890451 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:53.890583 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:53.898193 (+ 7610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:52:53.898199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:52:53.898202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:52:53.898240 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:52:53.900130 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:53.905388 (+ 5258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:53.905394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:53.905396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:53.905526 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:53.906123 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:53.906128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:53.906130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:53.906400 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:52:53.906496 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:53.907397 (+ 901us) server_negotiation.cc:299] Negotiation successful
0811 11:52:53.907568 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 11:52:53.909684 24505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:53.863567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:42831 (local address 127.3.221.66:44115)
0811 11:52:53.864237 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:53.864305 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:53.865530 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:53.866651 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:52:53.866662 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:52:53.867087 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:52:53.867869 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:53.867892 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:53.869112 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:53.869115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:53.886304 (+ 17189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:53.886334 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:53.890897 (+ 4563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:53.890903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:53.891201 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:53.895005 (+ 3804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:52:53.895035 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:52:53.897891 (+ 2856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:52:53.900311 (+ 2420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:53.900321 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:53.900345 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:53.900766 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:53.905677 (+ 4911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:53.905680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:53.905682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:53.905973 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:53.906585 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:52:53.906594 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:52:53.907043 (+ 449us) client_negotiation.cc:769] Sending connection context
0811 11:52:53.908042 (+ 999us) client_negotiation.cc:240] Negotiation successful
0811 11:52:53.908453 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":187,"threads_started":1}
I20260811 11:52:53.911247 24503 heartbeater.cc:347] Connected to a master server at 127.3.221.126:42831
I20260811 11:52:53.911540 24503 heartbeater.cc:464] Registering TS with master...
I20260811 11:52:53.912227 24503 heartbeater.cc:511] Master 127.3.221.126:42831 requested a full tablet report, sending...
I20260811 11:52:53.913534 23981 ts_manager.cc:194] Registered new tserver with Master: 5b9318b4d8314adfa368a3a85228fca5 (127.3.221.66:32805)
I20260811 11:52:53.914386 23981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:44115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
I20260811 11:52:53.926614 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.221.84:48725/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:42831
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:52:54.131876 24510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:52:54.132262 24510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:52:54.132337 24510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:52:54.139007 24510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:52:54.139256 24510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:52:54.139420 24510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:52:54.148185 24510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.221.84:48725/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:52:54.149926 24510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:52:54.151386 24510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:52:54.175906 24521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:52:54.176321 24522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:52:54.176601 24524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:52:54.177911 24510 server_base.cc:1034] running on GCE node
I20260811 11:52:54.178644 24510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:52:54.179527 24510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:52:54.183171 24510 hybrid_clock.cc:648] HybridClock initialized: now 1786449174183088 us; error 91 us; skew 500 ppm
Aug 11 11:52:54 dist-test-slave-bndk krb5kdc[23266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449173, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
Aug 11 11:52:54 dist-test-slave-bndk krb5kdc[23266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449174, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:52:54.187011 24510 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:52:54.188239 24510 webserver.cc:460] Webserver started at http://127.3.221.67:38383/ using document root <none> and password file <none>
I20260811 11:52:54.189055 24510 fs_manager.cc:359] Metadata directory not provided
I20260811 11:52:54.189210 24510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:52:54.189530 24510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 11:52:54 dist-test-slave-bndk krb5kdc[23266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449174, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
I20260811 11:52:54.211725 24510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "fcbfdc38a3ff4605a605895f17169728"
format_stamp: "Formatted at 2026-08-11 11:52:54 on dist-test-slave-bndk"
server_key: "5a10b36ad5270f9f97009f33a087dcfc"
server_key_iv: "9682cbee8fe3aba9d1f765a578d84e04"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:54.212710 24510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "fcbfdc38a3ff4605a605895f17169728"
format_stamp: "Formatted at 2026-08-11 11:52:54 on dist-test-slave-bndk"
server_key: "5a10b36ad5270f9f97009f33a087dcfc"
server_key_iv: "9682cbee8fe3aba9d1f765a578d84e04"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:54.218842 24510 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260811 11:52:54.241561 24534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:54.243499 24510 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260811 11:52:54.243675 24510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "fcbfdc38a3ff4605a605895f17169728"
format_stamp: "Formatted at 2026-08-11 11:52:54 on dist-test-slave-bndk"
server_key: "5a10b36ad5270f9f97009f33a087dcfc"
server_key_iv: "9682cbee8fe3aba9d1f765a578d84e04"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:54.243852 24510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:52:54.290010 24510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:52:54.294209 24510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:52:54.294500 24510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:52:54.295354 24510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:52:54.296592 24510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:52:54.296711 24510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:54.296823 24510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:52:54.296891 24510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:54.312929 24510 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:36225
I20260811 11:52:54.315310 24510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:52:54.323730 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 24510
I20260811 11:52:54.324117 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:52:54.324574 3957 external_mini_cluster.cc:1428] Setting key 703a9940ff0d25b5bd2ab5198aadf6d6
I20260811 11:52:54.316249 24704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:36225 every 8 connection(s)
Aug 11 11:52:54 dist-test-slave-bndk krb5kdc[23266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449174, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:52:54.346040 24302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:54.321020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:51787 (local address 127.3.221.126:42831)
0811 11:52:54.321157 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:54.321162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:54.326023 (+ 4861us) server_negotiation.cc:407] Connection header received
0811 11:52:54.332283 (+ 6260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:54.332288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:54.332370 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:54.332434 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:52:54.334660 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:54.335467 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:54.337376 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:54.337738 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:54.337840 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:54.341258 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:52:54.341262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:52:54.341265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:52:54.341300 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:52:54.343166 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:54.343952 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:54.343958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:54.343960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:54.344064 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:54.344517 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:54.344521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:54.344523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:54.344790 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:52:54.344876 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:54.345698 (+ 822us) server_negotiation.cc:299] Negotiation successful
0811 11:52:54.345870 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 11:52:54.348258 24708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:54.319788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:42831 (local address 127.3.221.67:51787)
0811 11:52:54.325745 (+ 5957us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:54.325833 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:54.327097 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:54.333095 (+ 5998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:52:54.333109 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:52:54.333591 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:52:54.334435 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:54.334461 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:54.335666 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:54.335669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:54.337175 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:54.337191 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:54.338099 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:54.338103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:54.338326 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:54.339103 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:52:54.339127 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:52:54.341010 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:52:54.343344 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:54.343352 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:54.343375 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:54.343813 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:54.344180 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:54.344183 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:54.344186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:54.344395 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:54.344950 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:52:54.344958 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:52:54.345373 (+ 415us) client_negotiation.cc:769] Sending connection context
0811 11:52:54.346722 (+ 1349us) client_negotiation.cc:240] Negotiation successful
0811 11:52:54.347161 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5641,"thread_start_us":162,"threads_started":1}
I20260811 11:52:54.349807 24706 heartbeater.cc:347] Connected to a master server at 127.3.221.126:42831
I20260811 11:52:54.350265 24706 heartbeater.cc:464] Registering TS with master...
I20260811 11:52:54.350950 24706 heartbeater.cc:511] Master 127.3.221.126:42831 requested a full tablet report, sending...
I20260811 11:52:54.353163 23981 ts_manager.cc:194] Registered new tserver with Master: fcbfdc38a3ff4605a605895f17169728 (127.3.221.67:36225)
I20260811 11:52:54.353936 23981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:51787
I20260811 11:52:54.365949 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:52:54.367352 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 24101
I20260811 11:52:54.383738 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 24305
I20260811 11:52:54.404109 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 24510
I20260811 11:52:54.416193 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 23930
I20260811 11:52:54.435453 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:42831
--webserver_interface=127.3.221.126
--webserver_port=34839
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:42831
--ranger_config_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.221.84:48725/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.221.84:48725/kms with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:52:54.557302 24709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:52:54.557631 24709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:52:54.557672 24709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:52:54.561818 24709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:52:54.561893 24709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:52:54.562093 24709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:52:54.562145 24709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:52:54.562176 24709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:52:54.568887 24709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.221.84:48725/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:42831
--ranger_config_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:42831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=34839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:52:54.570185 24709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:52:54.571314 24709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:52:54.581315 24720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:52:54.581509 24709 server_base.cc:1034] running on GCE node
W20260811 11:52:54.581444 24715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:52:54.581326 24716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:52:54.582378 24709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:52:54.583647 24709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:52:54.584834 24709 hybrid_clock.cc:648] HybridClock initialized: now 1786449174584814 us; error 37 us; skew 500 ppm
Aug 11 11:52:54 dist-test-slave-bndk krb5kdc[23266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449174, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:52:54.587998 24709 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:52:54.589118 24709 webserver.cc:460] Webserver started at http://127.3.221.126:34839/ using document root <none> and password file <none>
I20260811 11:52:54.589738 24709 fs_manager.cc:359] Metadata directory not provided
I20260811 11:52:54.589818 24709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 11:52:54 dist-test-slave-bndk krb5kdc[23266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449174, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
I20260811 11:52:54.666806 24709 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:52:54.669600 24734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:54.671041 24709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 11:52:54.671172 24709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "4cc67b9dac514daeb5175042e57f5cc4"
format_stamp: "Formatted at 2026-08-11 11:52:52 on dist-test-slave-bndk"
server_key: "b1ac482b5bd8814dfd0b7b0c7f2020e3"
server_key_iv: "611c2e7242459226cacca2757f794f6c"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:54.671795 24709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:52:54.683527 24709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:52:54.686702 24709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:52:54.686910 24709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:52:54.694494 24709 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:42831
I20260811 11:52:54.694527 24817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:42831 every 8 connection(s)
I20260811 11:52:54.695683 24709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:52:54.700937 24820 sys_catalog.cc:263] Verifying existing consensus state
I20260811 11:52:54.702869 24820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4: Bootstrap starting.
I20260811 11:52:54.704923 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 24709
I20260811 11:52:54.705416 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.221.84:48725/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:33277
--local_ip_for_outbound_sockets=127.3.221.65
--tserver_master_addrs=127.3.221.126:42831
--webserver_port=36889
--webserver_interface=127.3.221.65
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.221.84:48725/kms with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:52:54.719985 24820 log.cc:824] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4: Log is configured to *not* fsync() on all Append() calls
I20260811 11:52:54.727630 24820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:52:54.728045 24820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4: Bootstrap complete.
I20260811 11:52:54.732213 24820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc67b9dac514daeb5175042e57f5cc4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42831 } }
I20260811 11:52:54.732651 24820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc67b9dac514daeb5175042e57f5cc4, State: Initialized, Role: FOLLOWER
I20260811 11:52:54.733155 24820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc67b9dac514daeb5175042e57f5cc4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42831 } }
I20260811 11:52:54.733332 24820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:52:54.733407 24820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:52:54.733508 24820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [term 1 FOLLOWER]: Advancing to term 2
I20260811 11:52:54.735605 24820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc67b9dac514daeb5175042e57f5cc4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42831 } }
I20260811 11:52:54.735958 24820 leader_election.cc:302] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cc67b9dac514daeb5175042e57f5cc4; no voters:
I20260811 11:52:54.736284 24820 leader_election.cc:288] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 11:52:54.736393 24827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 11:52:54.736603 24827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [term 2 LEADER]: Becoming Leader. State: Replica: 4cc67b9dac514daeb5175042e57f5cc4, State: Running, Role: LEADER
I20260811 11:52:54.736934 24827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc67b9dac514daeb5175042e57f5cc4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42831 } }
I20260811 11:52:54.737675 24820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:52:54.738428 24828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4cc67b9dac514daeb5175042e57f5cc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc67b9dac514daeb5175042e57f5cc4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42831 } } }
I20260811 11:52:54.738579 24828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [sys.catalog]: This master's current role is: LEADER
I20260811 11:52:54.738988 24829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cc67b9dac514daeb5175042e57f5cc4. Latest consensus state: current_term: 2 leader_uuid: "4cc67b9dac514daeb5175042e57f5cc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc67b9dac514daeb5175042e57f5cc4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 42831 } } }
I20260811 11:52:54.739130 24829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4 [sys.catalog]: This master's current role is: LEADER
I20260811 11:52:54.739796 24840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:52:54.740142 24820 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260811 11:52:54.746803 24840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:52:54.747766 24840 catalog_manager.cc:1259] Loaded cluster ID: c115707693d04cd5ba296482a9eabf3a
I20260811 11:52:54.747884 24840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:52:54.750977 24840 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:52:54.751904 24840 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4: Loaded TSK: 0
I20260811 11:52:54.753507 24840 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 11:52:54.872354 24824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:52:54.872790 24824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:52:54.872865 24824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:52:54.879242 24824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:52:54.879339 24824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:52:54.879447 24824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:52:54.887742 24824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.221.84:48725/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:33277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=36889
--tserver_master_addrs=127.3.221.126:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:52:54.889324 24824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:52:54.890816 24824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:52:54.904122 24873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:52:54.904119 24875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:52:54.904233 24877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:52:54.905315 24824 server_base.cc:1034] running on GCE node
I20260811 11:52:54.905817 24824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:52:54.906622 24824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:52:54.909999 24824 hybrid_clock.cc:648] HybridClock initialized: now 1786449174909943 us; error 65 us; skew 500 ppm
Aug 11 11:52:54 dist-test-slave-bndk krb5kdc[23266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449174, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:52:54.914275 24824 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:52:54.915673 24824 webserver.cc:460] Webserver started at http://127.3.221.65:36889/ using document root <none> and password file <none>
I20260811 11:52:54.916460 24824 fs_manager.cc:359] Metadata directory not provided
I20260811 11:52:54.916667 24824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 11:52:54 dist-test-slave-bndk krb5kdc[23266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449174, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
I20260811 11:52:54.936446 24824 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260811 11:52:54.940991 24892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:54.942879 24824 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260811 11:52:54.943104 24824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "09f190a507c34a28b35fc407add36bfd"
format_stamp: "Formatted at 2026-08-11 11:52:53 on dist-test-slave-bndk"
server_key: "40d758632086ebc80e4c5062fa6dc497"
server_key_iv: "279a0490011eccab140c4f704f07e418"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:54.943956 24824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:52:54.963404 24824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:52:54.966692 24824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:52:54.966908 24824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:52:54.967556 24824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:52:54.968573 24824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:52:54.968636 24824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:54.968699 24824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:52:54.968722 24824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:54.980249 24824 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:33277
I20260811 11:52:54.982113 24824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:52:54.985591 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 24824
I20260811 11:52:54.986102 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.221.84:48725/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:32805
--local_ip_for_outbound_sockets=127.3.221.66
--tserver_master_addrs=127.3.221.126:42831
--webserver_port=39437
--webserver_interface=127.3.221.66
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.221.84:48725/kms with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:52:54.989758 25058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:33277 every 8 connection(s)
Aug 11 11:52:55 dist-test-slave-bndk krb5kdc[23266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449174, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:52:55.008930 25063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:54.989767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:42831 (local address 127.3.221.65:37949)
0811 11:52:54.990469 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:54.990535 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:54.991685 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:54.993887 (+ 2202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:52:54.993897 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:52:54.994301 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:52:54.995021 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:54.995040 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:54.996639 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:54.996641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:54.998034 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:54.998049 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:54.999481 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:54.999486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:54.999696 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:55.000448 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:52:55.000474 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:52:55.002212 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:52:55.004957 (+ 2745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:55.004965 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:55.004981 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:55.005377 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:55.005858 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:55.005861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:55.005863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:55.006044 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:55.006872 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:52:55.006879 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:52:55.007192 (+ 313us) client_negotiation.cc:769] Sending connection context
0811 11:52:55.007394 (+ 202us) client_negotiation.cc:240] Negotiation successful
0811 11:52:55.007750 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":206,"threads_started":1}
I20260811 11:52:55.008929 25064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:54.991702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:37949 (local address 127.3.221.126:42831)
0811 11:52:54.992273 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:54.992394 (+ 121us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:54.992448 (+ 54us) server_negotiation.cc:407] Connection header received
0811 11:52:54.992792 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:54.992810 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:54.993323 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:54.995252 (+ 1929us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:52:54.995498 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:54.996480 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:54.998207 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:54.999011 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:54.999206 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:55.002514 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:52:55.002522 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:52:55.002545 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:52:55.002584 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:52:55.004794 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:55.005545 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:55.005553 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:55.005559 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:55.005740 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:55.006160 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:55.006164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:55.006166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:55.006740 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:52:55.006907 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:55.007584 (+ 677us) server_negotiation.cc:299] Negotiation successful
0811 11:52:55.007932 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":137,"threads_started":1}
I20260811 11:52:55.012634 25059 heartbeater.cc:347] Connected to a master server at 127.3.221.126:42831
I20260811 11:52:55.012957 25059 heartbeater.cc:464] Registering TS with master...
I20260811 11:52:55.013625 25059 heartbeater.cc:511] Master 127.3.221.126:42831 requested a full tablet report, sending...
W20260811 11:52:55.016160 25059 heartbeater.cc:644] Failed to heartbeat to 127.3.221.126:42831 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 11:52:55.016302 25059 heartbeater.cc:464] Registering TS with master...
I20260811 11:52:55.016505 25059 heartbeater.cc:511] Master 127.3.221.126:42831 requested a full tablet report, sending...
I20260811 11:52:55.017449 25059 heartbeater.cc:464] Registering TS with master...
I20260811 11:52:55.017639 25059 heartbeater.cc:511] Master 127.3.221.126:42831 requested a full tablet report, sending...
W20260811 11:52:55.018416 25059 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 11:52:55.136767 25062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:52:55.137064 25062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:52:55.137111 25062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:52:55.141856 25062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:52:55.141939 25062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:52:55.142024 25062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:52:55.148029 25062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.221.84:48725/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:32805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=39437
--tserver_master_addrs=127.3.221.126:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:52:55.149505 25062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:52:55.150732 25062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:52:55.161671 25074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:52:55.161707 25073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:52:55.162025 25062 server_base.cc:1034] running on GCE node
W20260811 11:52:55.162320 25077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:52:55.162993 25062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:52:55.163765 25062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:52:55.164975 25062 hybrid_clock.cc:648] HybridClock initialized: now 1786449175164944 us; error 42 us; skew 500 ppm
Aug 11 11:52:55 dist-test-slave-bndk krb5kdc[23266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449175, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:52:55.169171 25062 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:52:55.170833 25062 webserver.cc:460] Webserver started at http://127.3.221.66:39437/ using document root <none> and password file <none>
I20260811 11:52:55.171877 25062 fs_manager.cc:359] Metadata directory not provided
I20260811 11:52:55.172005 25062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 11:52:55 dist-test-slave-bndk krb5kdc[23266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449175, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
I20260811 11:52:55.204134 25062 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 11:52:55.207770 25089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:55.209585 25062 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260811 11:52:55.209755 25062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "5b9318b4d8314adfa368a3a85228fca5"
format_stamp: "Formatted at 2026-08-11 11:52:53 on dist-test-slave-bndk"
server_key: "febe65b4953a0d4024670993d92d6233"
server_key_iv: "b0431f44654b65182bfd5a14315544f1"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:55.210549 25062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:52:55.231197 25062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:52:55.235848 25062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:52:55.236080 25062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:52:55.236860 25062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:52:55.238205 25062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:52:55.238272 25062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:55.238332 25062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:52:55.238360 25062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:55.251372 25062 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:32805
I20260811 11:52:55.251420 25271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:32805 every 8 connection(s)
I20260811 11:52:55.253367 25062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:52:55.259824 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 25062
I20260811 11:52:55.260484 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.221.84:48725/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:36225
--local_ip_for_outbound_sockets=127.3.221.67
--tserver_master_addrs=127.3.221.126:42831
--webserver_port=38383
--webserver_interface=127.3.221.67
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:42277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.221.84:48725/kms with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 11:52:55 dist-test-slave-bndk krb5kdc[23266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449175, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:52:55.276542 25064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:55.257797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:42659 (local address 127.3.221.126:42831)
0811 11:52:55.258002 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:55.258007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:55.262854 (+ 4847us) server_negotiation.cc:407] Connection header received
0811 11:52:55.263473 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:55.263477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:55.263542 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:55.263623 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:52:55.265618 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:55.266468 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:55.268386 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:55.268739 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:55.268847 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:55.272526 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:52:55.272531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:52:55.272534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:52:55.272573 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:52:55.274189 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:55.274798 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:55.274801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:55.274802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:55.274886 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:55.275278 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:55.275282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:55.275283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:55.275504 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:52:55.275598 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:55.276214 (+ 616us) server_negotiation.cc:299] Negotiation successful
0811 11:52:55.276366 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260811 11:52:55.277601 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:55.256455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:42831 (local address 127.3.221.66:42659)
0811 11:52:55.261938 (+ 5483us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:55.262016 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:55.263228 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:55.264016 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:52:55.264028 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:52:55.264543 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:52:55.265401 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:55.265429 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:55.266695 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:55.266698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:55.268215 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:55.268230 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:55.269321 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:55.269327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:55.269564 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:55.270389 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:52:55.270417 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:52:55.272261 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:52:55.274343 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:55.274352 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:55.274373 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:55.274669 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:55.274980 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:55.274983 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:55.274984 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:55.275151 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:55.275617 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:52:55.275624 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:52:55.275915 (+ 291us) client_negotiation.cc:769] Sending connection context
0811 11:52:55.276105 (+ 190us) client_negotiation.cc:240] Negotiation successful
0811 11:52:55.276566 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5171,"thread_start_us":149,"threads_started":1}
I20260811 11:52:55.280329 25272 heartbeater.cc:347] Connected to a master server at 127.3.221.126:42831
I20260811 11:52:55.280627 25272 heartbeater.cc:464] Registering TS with master...
I20260811 11:52:55.281585 25272 heartbeater.cc:511] Master 127.3.221.126:42831 requested a full tablet report, sending...
W20260811 11:52:55.283807 25272 heartbeater.cc:644] Failed to heartbeat to 127.3.221.126:42831 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 11:52:55.283970 25272 heartbeater.cc:464] Registering TS with master...
I20260811 11:52:55.284195 25272 heartbeater.cc:511] Master 127.3.221.126:42831 requested a full tablet report, sending...
I20260811 11:52:55.286510 25272 heartbeater.cc:464] Registering TS with master...
I20260811 11:52:55.286775 25272 heartbeater.cc:511] Master 127.3.221.126:42831 requested a full tablet report, sending...
W20260811 11:52:55.287839 25272 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 11:52:55.413029 25276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:52:55.413403 25276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:52:55.413482 25276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:52:55.417819 25276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:52:55.417891 25276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:52:55.417972 25276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:52:55.424606 25276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:42277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.221.84:48725/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:36225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=38383
--tserver_master_addrs=127.3.221.126:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:52:55.425899 25276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:52:55.427064 25276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:52:55.440814 25283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:52:55.440831 25285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:52:55.440769 25287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:52:55.441447 25276 server_base.cc:1034] running on GCE node
I20260811 11:52:55.442085 25276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:52:55.443030 25276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:52:55.450562 25276 hybrid_clock.cc:648] HybridClock initialized: now 1786449175450499 us; error 83 us; skew 500 ppm
Aug 11 11:52:55 dist-test-slave-bndk krb5kdc[23266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449175, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:52:55.454982 25276 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:52:55.456384 25276 webserver.cc:460] Webserver started at http://127.3.221.67:38383/ using document root <none> and password file <none>
I20260811 11:52:55.457373 25276 fs_manager.cc:359] Metadata directory not provided
I20260811 11:52:55.457520 25276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 11:52:55 dist-test-slave-bndk krb5kdc[23266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449175, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
I20260811 11:52:55.488592 25276 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.000s sys 0.004s
I20260811 11:52:55.492249 25303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:55.495432 25276 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260811 11:52:55.495604 25276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "fcbfdc38a3ff4605a605895f17169728"
format_stamp: "Formatted at 2026-08-11 11:52:54 on dist-test-slave-bndk"
server_key: "5a10b36ad5270f9f97009f33a087dcfc"
server_key_iv: "9682cbee8fe3aba9d1f765a578d84e04"
server_key_version: "kuduclusterkey@0"
I20260811 11:52:55.496407 25276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:52:55.512257 25276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:52:55.516031 25276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:52:55.516207 25276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:52:55.517498 25276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:52:55.518994 25276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:52:55.519090 25276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:55.519168 25276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:52:55.519200 25276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:55.532204 25276 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:36225
I20260811 11:52:55.533795 25276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:52:55.536314 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 25276
I20260811 11:52:55.534752 25473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:36225 every 8 connection(s)
I20260811 11:52:55.550117 25064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:55.538948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46638 (local address 127.3.221.126:42831)
0811 11:52:55.539106 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:55.539110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:55.539128 (+ 18us) server_negotiation.cc:407] Connection header received
0811 11:52:55.539181 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:55.539184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:55.539239 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:55.539333 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:52:55.540676 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:55.541551 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:55.542850 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:55.543207 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:55.543286 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:55.546008 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:52:55.546012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:52:55.546015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:52:55.546049 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:52:55.547797 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:55.548537 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:55.548541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:55.548543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:55.548671 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:55.549015 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:55.549018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:55.549020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:55.549286 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:52:55.549514 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:55.549796 (+ 282us) server_negotiation.cc:299] Negotiation successful
0811 11:52:55.549937 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Aug 11 11:52:55 dist-test-slave-bndk krb5kdc[23266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449175, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:52:55.558318 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:55.541850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:35813 (local address 127.3.221.126:42831)
0811 11:52:55.545444 (+ 3594us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:55.545449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:55.545465 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:52:55.546348 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:55.546351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:55.546415 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:55.546510 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:52:55.548252 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:55.548943 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:55.550923 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:55.551166 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:55.551306 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:55.554536 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:52:55.554539 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:52:55.554541 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:52:55.554567 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:52:55.555952 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:55.556519 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:55.556522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:55.556523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:55.556602 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:55.556971 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:55.556973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:55.556975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:55.557182 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:52:55.557301 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:55.557981 (+ 680us) server_negotiation.cc:299] Negotiation successful
0811 11:52:55.558131 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3512,"thread_start_us":92,"threads_started":1}
I20260811 11:52:55.559245 25479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:55.540639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:42831 (local address 127.3.221.67:35813)
0811 11:52:55.544968 (+ 4329us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:55.545038 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:55.546065 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:55.546966 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:52:55.546976 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:52:55.547335 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:52:55.548090 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:55.548109 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:55.549088 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:55.549091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:55.550781 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:55.550795 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:55.551479 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:55.551484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:55.551699 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:55.552681 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:52:55.552704 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:52:55.554329 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:52:55.556082 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:55.556090 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:55.556113 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:55.556404 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:55.556689 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:55.556692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:55.556694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:55.556865 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:55.557302 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:52:55.557311 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:52:55.557599 (+ 288us) client_negotiation.cc:769] Sending connection context
0811 11:52:55.557879 (+ 280us) client_negotiation.cc:240] Negotiation successful
0811 11:52:55.558256 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3967,"thread_start_us":172,"threads_started":1}
I20260811 11:52:55.560690 25474 heartbeater.cc:347] Connected to a master server at 127.3.221.126:42831
I20260811 11:52:55.560985 25474 heartbeater.cc:464] Registering TS with master...
I20260811 11:52:55.561648 25474 heartbeater.cc:511] Master 127.3.221.126:42831 requested a full tablet report, sending...
W20260811 11:52:55.565934 25474 heartbeater.cc:644] Failed to heartbeat to 127.3.221.126:42831 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 11:52:55.566075 25474 heartbeater.cc:464] Registering TS with master...
I20260811 11:52:55.566293 25474 heartbeater.cc:511] Master 127.3.221.126:42831 requested a full tablet report, sending...
I20260811 11:52:55.567418 25474 heartbeater.cc:464] Registering TS with master...
I20260811 11:52:55.567652 25474 heartbeater.cc:511] Master 127.3.221.126:42831 requested a full tablet report, sending...
W20260811 11:52:55.568596 25474 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 11 11:52:55 dist-test-slave-bndk krb5kdc[23266](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786449175, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:52:56.018688 25059 heartbeater.cc:464] Registering TS with master...
I20260811 11:52:56.019045 25059 heartbeater.cc:511] Master 127.3.221.126:42831 requested a full tablet report, sending...
I20260811 11:52:56.020495 24759 ts_manager.cc:194] Registered new tserver with Master: 09f190a507c34a28b35fc407add36bfd (127.3.221.65:33277)
I20260811 11:52:56.022351 24759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:37949
I20260811 11:52:56.288091 25272 heartbeater.cc:464] Registering TS with master...
I20260811 11:52:56.288395 25272 heartbeater.cc:511] Master 127.3.221.126:42831 requested a full tablet report, sending...
I20260811 11:52:56.289183 24759 ts_manager.cc:194] Registered new tserver with Master: 5b9318b4d8314adfa368a3a85228fca5 (127.3.221.66:32805)
I20260811 11:52:56.290023 24759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:42659
Aug 11 11:52:56 dist-test-slave-bndk krb5kdc[23266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449175, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
I20260811 11:52:56.568882 25474 heartbeater.cc:464] Registering TS with master...
I20260811 11:52:56.569320 25474 heartbeater.cc:511] Master 127.3.221.126:42831 requested a full tablet report, sending...
I20260811 11:52:56.570289 24759 ts_manager.cc:194] Registered new tserver with Master: fcbfdc38a3ff4605a605895f17169728 (127.3.221.67:36225)
I20260811 11:52:56.571056 24759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:35813
I20260811 11:52:56.573911 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 11:52:56 dist-test-slave-bndk krb5kdc[23266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449175, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for HTTP/127.3.221.84@KRBTEST.COM
I20260811 11:52:56.589313 25513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:56.578004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46652 (local address 127.3.221.126:42831)
0811 11:52:56.578439 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:56.578444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:56.578664 (+ 220us) server_negotiation.cc:407] Connection header received
0811 11:52:56.578868 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:56.578872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:56.578923 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:56.578982 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:52:56.579980 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.580789 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.581993 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.582454 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.582584 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:56.584431 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:52:56.584435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:52:56.584438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:52:56.584471 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:52:56.586871 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:56.587412 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:56.587415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:56.587417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:56.587513 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:56.587811 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:56.587814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:56.587816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:56.588048 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:52:56.588170 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:56.588950 (+ 780us) server_negotiation.cc:299] Negotiation successful
0811 11:52:56.589090 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":106,"threads_started":1}
I20260811 11:52:56.594367 24759 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:52:56.599479 24759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:52:56.639647 25520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:56.629502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:36225 (local address 127.0.0.1:60658)
0811 11:52:56.633686 (+ 4184us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:56.633700 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:56.633803 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:56.634723 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:52:56.634731 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:52:56.634760 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:52:56.634923 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:56.634931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.636188 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.636192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:56.637643 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:56.637655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.639007 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.639011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:56.639138 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:56.639185 (+ 47us) client_negotiation.cc:769] Sending connection context
0811 11:52:56.639309 (+ 124us) client_negotiation.cc:240] Negotiation successful
0811 11:52:56.639473 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4115,"thread_start_us":104,"threads_started":1}
I20260811 11:52:56.640316 25523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:56.629939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.3.221.67:36225)
0811 11:52:56.633532 (+ 3593us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:56.633540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:56.633907 (+ 367us) server_negotiation.cc:407] Connection header received
0811 11:52:56.634010 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:56.634016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:56.634287 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:56.634485 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:52:56.635076 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.636039 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.637782 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.638637 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.638723 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:56.638892 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:56.640039 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0811 11:52:56.640168 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3214,"thread_start_us":88,"threads_started":1}
I20260811 11:52:56.644255 25524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:56.630825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:33277 (local address 127.0.0.1:58204)
0811 11:52:56.636980 (+ 6155us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:56.636996 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:56.637097 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:56.638620 (+ 1523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:52:56.638623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:52:56.638643 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:52:56.638745 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:56.638751 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.640573 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.640576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:56.641730 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:56.641742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.643831 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.643835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:56.643936 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:56.643963 (+ 27us) client_negotiation.cc:769] Sending connection context
0811 11:52:56.644034 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 11:52:56.644110 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6083,"thread_start_us":128,"threads_started":1}
I20260811 11:52:56.645033 25525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:56.631345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58204 (local address 127.3.221.65:33277)
0811 11:52:56.636815 (+ 5470us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:56.636823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:56.637868 (+ 1045us) server_negotiation.cc:407] Connection header received
0811 11:52:56.637973 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:56.637979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:56.638249 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:56.638455 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:52:56.639257 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.640430 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.642288 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.643101 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.643218 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:56.643394 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:56.644774 (+ 1380us) server_negotiation.cc:299] Negotiation successful
0811 11:52:56.644893 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5336,"thread_start_us":92,"threads_started":1}
I20260811 11:52:56.645291 25377 tablet_service.cc:1467] Processing CreateTablet for tablet d117b40d18fa4c32bfebb3491048c79d (DEFAULT_TABLE table=test-table [id=6f39cd54ee2f4a2baa12a6860ad85d7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:52:56.646879 25377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d117b40d18fa4c32bfebb3491048c79d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:52:56.649936 24960 tablet_service.cc:1467] Processing CreateTablet for tablet d117b40d18fa4c32bfebb3491048c79d (DEFAULT_TABLE table=test-table [id=6f39cd54ee2f4a2baa12a6860ad85d7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:52:56.651211 24960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d117b40d18fa4c32bfebb3491048c79d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:52:56.659024 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:56.628370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:32805 (local address 127.0.0.1:36278)
0811 11:52:56.632680 (+ 4310us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:56.632754 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:56.632941 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:56.642743 (+ 9802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:52:56.642746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:52:56.642771 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:52:56.642872 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:56.642879 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.644859 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.644862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:56.645961 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:56.645974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.658525 (+ 12551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.658530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:56.658648 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:56.658682 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 11:52:56.658735 (+ 53us) client_negotiation.cc:240] Negotiation successful
0811 11:52:56.658840 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4080,"thread_start_us":111,"threads_started":1}
I20260811 11:52:56.659864 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:56.628488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36278 (local address 127.3.221.66:32805)
0811 11:52:56.640942 (+ 12454us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:56.640950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:56.640980 (+ 30us) server_negotiation.cc:407] Connection header received
0811 11:52:56.641070 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:56.641076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:56.641361 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:56.643047 (+ 1686us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:52:56.643206 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.644724 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.657428 (+ 12704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.658344 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.659368 (+ 1024us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:56.659541 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:56.659598 (+ 57us) server_negotiation.cc:299] Negotiation successful
0811 11:52:56.659704 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12320,"thread_start_us":74,"threads_started":1}
I20260811 11:52:56.663416 25165 tablet_service.cc:1467] Processing CreateTablet for tablet d117b40d18fa4c32bfebb3491048c79d (DEFAULT_TABLE table=test-table [id=6f39cd54ee2f4a2baa12a6860ad85d7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:52:56.664958 25165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d117b40d18fa4c32bfebb3491048c79d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:52:56.669768 25528 tablet_bootstrap.cc:492] T d117b40d18fa4c32bfebb3491048c79d P fcbfdc38a3ff4605a605895f17169728: Bootstrap starting.
I20260811 11:52:56.672535 25528 tablet_bootstrap.cc:654] T d117b40d18fa4c32bfebb3491048c79d P fcbfdc38a3ff4605a605895f17169728: Neither blocks nor log segments found. Creating new log.
I20260811 11:52:56.673576 25528 log.cc:824] T d117b40d18fa4c32bfebb3491048c79d P fcbfdc38a3ff4605a605895f17169728: Log is configured to *not* fsync() on all Append() calls
I20260811 11:52:56.679734 25529 tablet_bootstrap.cc:492] T d117b40d18fa4c32bfebb3491048c79d P 09f190a507c34a28b35fc407add36bfd: Bootstrap starting.
I20260811 11:52:56.682830 25529 tablet_bootstrap.cc:654] T d117b40d18fa4c32bfebb3491048c79d P 09f190a507c34a28b35fc407add36bfd: Neither blocks nor log segments found. Creating new log.
I20260811 11:52:56.684350 25529 log.cc:824] T d117b40d18fa4c32bfebb3491048c79d P 09f190a507c34a28b35fc407add36bfd: Log is configured to *not* fsync() on all Append() calls
I20260811 11:52:56.685575 25528 tablet_bootstrap.cc:492] T d117b40d18fa4c32bfebb3491048c79d P fcbfdc38a3ff4605a605895f17169728: No bootstrap required, opened a new log
I20260811 11:52:56.685845 25528 ts_tablet_manager.cc:1397] T d117b40d18fa4c32bfebb3491048c79d P fcbfdc38a3ff4605a605895f17169728: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20260811 11:52:56.694672 25528 raft_consensus.cc:348] T d117b40d18fa4c32bfebb3491048c79d P fcbfdc38a3ff4605a605895f17169728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9318b4d8314adfa368a3a85228fca5" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 32805 } } peers { permanent_uuid: "fcbfdc38a3ff4605a605895f17169728" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 36225 } } peers { permanent_uuid: "09f190a507c34a28b35fc407add36bfd" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 33277 } }
I20260811 11:52:56.695091 25528 raft_consensus.cc:374] T d117b40d18fa4c32bfebb3491048c79d P fcbfdc38a3ff4605a605895f17169728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:52:56.695158 25528 raft_consensus.cc:729] T d117b40d18fa4c32bfebb3491048c79d P fcbfdc38a3ff4605a605895f17169728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcbfdc38a3ff4605a605895f17169728, State: Initialized, Role: FOLLOWER
I20260811 11:52:56.695714 25528 consensus_queue.cc:260] T d117b40d18fa4c32bfebb3491048c79d P fcbfdc38a3ff4605a605895f17169728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9318b4d8314adfa368a3a85228fca5" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 32805 } } peers { permanent_uuid: "fcbfdc38a3ff4605a605895f17169728" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 36225 } } peers { permanent_uuid: "09f190a507c34a28b35fc407add36bfd" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 33277 } }
I20260811 11:52:56.696877 25528 ts_tablet_manager.cc:1428] T d117b40d18fa4c32bfebb3491048c79d P fcbfdc38a3ff4605a605895f17169728: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260811 11:52:56.697537 25474 heartbeater.cc:503] Master 127.3.221.126:42831 was elected leader, sending a full tablet report...
I20260811 11:52:56.697634 25531 tablet_bootstrap.cc:492] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5: Bootstrap starting.
I20260811 11:52:56.700356 25531 tablet_bootstrap.cc:654] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5: Neither blocks nor log segments found. Creating new log.
I20260811 11:52:56.701184 25531 log.cc:824] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5: Log is configured to *not* fsync() on all Append() calls
I20260811 11:52:56.714762 25529 tablet_bootstrap.cc:492] T d117b40d18fa4c32bfebb3491048c79d P 09f190a507c34a28b35fc407add36bfd: No bootstrap required, opened a new log
I20260811 11:52:56.715051 25529 ts_tablet_manager.cc:1397] T d117b40d18fa4c32bfebb3491048c79d P 09f190a507c34a28b35fc407add36bfd: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.000s
I20260811 11:52:56.721325 25529 raft_consensus.cc:348] T d117b40d18fa4c32bfebb3491048c79d P 09f190a507c34a28b35fc407add36bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9318b4d8314adfa368a3a85228fca5" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 32805 } } peers { permanent_uuid: "fcbfdc38a3ff4605a605895f17169728" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 36225 } } peers { permanent_uuid: "09f190a507c34a28b35fc407add36bfd" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 33277 } }
I20260811 11:52:56.721859 25529 raft_consensus.cc:374] T d117b40d18fa4c32bfebb3491048c79d P 09f190a507c34a28b35fc407add36bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:52:56.721971 25529 raft_consensus.cc:729] T d117b40d18fa4c32bfebb3491048c79d P 09f190a507c34a28b35fc407add36bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f190a507c34a28b35fc407add36bfd, State: Initialized, Role: FOLLOWER
I20260811 11:52:56.722671 25529 consensus_queue.cc:260] T d117b40d18fa4c32bfebb3491048c79d P 09f190a507c34a28b35fc407add36bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9318b4d8314adfa368a3a85228fca5" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 32805 } } peers { permanent_uuid: "fcbfdc38a3ff4605a605895f17169728" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 36225 } } peers { permanent_uuid: "09f190a507c34a28b35fc407add36bfd" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 33277 } }
I20260811 11:52:56.724139 25529 ts_tablet_manager.cc:1428] T d117b40d18fa4c32bfebb3491048c79d P 09f190a507c34a28b35fc407add36bfd: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260811 11:52:56.724648 25059 heartbeater.cc:503] Master 127.3.221.126:42831 was elected leader, sending a full tablet report...
I20260811 11:52:56.730499 25531 tablet_bootstrap.cc:492] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5: No bootstrap required, opened a new log
I20260811 11:52:56.730772 25531 ts_tablet_manager.cc:1397] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.000s
W20260811 11:52:56.734728 25060 tablet.cc:2369] T d117b40d18fa4c32bfebb3491048c79d P 09f190a507c34a28b35fc407add36bfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:52:56.735606 25531 raft_consensus.cc:348] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9318b4d8314adfa368a3a85228fca5" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 32805 } } peers { permanent_uuid: "fcbfdc38a3ff4605a605895f17169728" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 36225 } } peers { permanent_uuid: "09f190a507c34a28b35fc407add36bfd" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 33277 } }
I20260811 11:52:56.735962 25531 raft_consensus.cc:374] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:52:56.736021 25531 raft_consensus.cc:729] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b9318b4d8314adfa368a3a85228fca5, State: Initialized, Role: FOLLOWER
I20260811 11:52:56.736624 25531 consensus_queue.cc:260] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9318b4d8314adfa368a3a85228fca5" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 32805 } } peers { permanent_uuid: "fcbfdc38a3ff4605a605895f17169728" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 36225 } } peers { permanent_uuid: "09f190a507c34a28b35fc407add36bfd" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 33277 } }
I20260811 11:52:56.738042 25531 ts_tablet_manager.cc:1428] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260811 11:52:56.738489 25272 heartbeater.cc:503] Master 127.3.221.126:42831 was elected leader, sending a full tablet report...
W20260811 11:52:56.763072 25274 tablet.cc:2369] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:52:56.792600 25477 tablet.cc:2369] T d117b40d18fa4c32bfebb3491048c79d P fcbfdc38a3ff4605a605895f17169728: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:52:56.923106 25537 raft_consensus.cc:482] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:52:56.923393 25537 raft_consensus.cc:504] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9318b4d8314adfa368a3a85228fca5" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 32805 } } peers { permanent_uuid: "fcbfdc38a3ff4605a605895f17169728" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 36225 } } peers { permanent_uuid: "09f190a507c34a28b35fc407add36bfd" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 33277 } }
I20260811 11:52:56.924564 25537 leader_election.cc:288] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcbfdc38a3ff4605a605895f17169728 (127.3.221.67:36225), 09f190a507c34a28b35fc407add36bfd (127.3.221.65:33277)
I20260811 11:52:56.929709 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:56.924920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:36225 (local address 127.3.221.66:40573)
0811 11:52:56.925267 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:56.925300 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:56.925491 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:56.925899 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:52:56.925901 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:52:56.925964 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:52:56.926121 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:56.926127 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.926967 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.926970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:56.928131 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:56.928144 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.929180 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.929183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:56.929313 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:56.929341 (+ 28us) client_negotiation.cc:769] Sending connection context
0811 11:52:56.929414 (+ 73us) client_negotiation.cc:240] Negotiation successful
0811 11:52:56.929497 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"spinlock_wait_cycles":20224,"thread_start_us":133,"threads_started":1}
I20260811 11:52:56.929977 25525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:56.925563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:33175 (local address 127.3.221.65:33277)
0811 11:52:56.925707 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:56.925713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:56.925904 (+ 191us) server_negotiation.cc:407] Connection header received
0811 11:52:56.926016 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:56.926019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:56.926073 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:56.926159 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:52:56.926404 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.927124 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.928627 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.929256 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.929365 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:56.929420 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:56.929765 (+ 345us) server_negotiation.cc:299] Negotiation successful
0811 11:52:56.929852 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 11:52:56.930343 25523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:56.925021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:40573 (local address 127.3.221.67:36225)
0811 11:52:56.925151 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:56.925156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:56.925398 (+ 242us) server_negotiation.cc:407] Connection header received
0811 11:52:56.925628 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:56.925632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:56.925686 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:56.925737 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:52:56.926255 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.926844 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.928297 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.928883 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.929003 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:56.930070 (+ 1067us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:56.930135 (+ 65us) server_negotiation.cc:299] Negotiation successful
0811 11:52:56.930215 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 11:52:56.930593 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:56.925387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:33277 (local address 127.3.221.66:33175)
0811 11:52:56.925807 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:56.925820 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:56.925918 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:56.926177 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:52:56.926179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:52:56.926198 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:52:56.926301 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:56.926307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.927277 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.927281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:56.928471 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:56.928484 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.929519 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.929523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:56.929626 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:56.929649 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 11:52:56.930370 (+ 721us) client_negotiation.cc:240] Negotiation successful
0811 11:52:56.930452 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":100,"threads_started":1}
I20260811 11:52:56.931334 24964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d117b40d18fa4c32bfebb3491048c79d" candidate_uuid: "5b9318b4d8314adfa368a3a85228fca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f190a507c34a28b35fc407add36bfd" is_pre_election: true
I20260811 11:52:56.931334 25405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d117b40d18fa4c32bfebb3491048c79d" candidate_uuid: "5b9318b4d8314adfa368a3a85228fca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcbfdc38a3ff4605a605895f17169728" is_pre_election: true
I20260811 11:52:56.931690 24964 raft_consensus.cc:2454] T d117b40d18fa4c32bfebb3491048c79d P 09f190a507c34a28b35fc407add36bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b9318b4d8314adfa368a3a85228fca5 in term 0.
I20260811 11:52:56.931690 25405 raft_consensus.cc:2454] T d117b40d18fa4c32bfebb3491048c79d P fcbfdc38a3ff4605a605895f17169728 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b9318b4d8314adfa368a3a85228fca5 in term 0.
I20260811 11:52:56.932361 25093 leader_election.cc:302] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b9318b4d8314adfa368a3a85228fca5, fcbfdc38a3ff4605a605895f17169728; no voters:
I20260811 11:52:56.932651 25537 raft_consensus.cc:2781] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:52:56.932739 25537 raft_consensus.cc:482] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:52:56.932790 25537 raft_consensus.cc:3037] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:52:56.934249 25537 raft_consensus.cc:504] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9318b4d8314adfa368a3a85228fca5" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 32805 } } peers { permanent_uuid: "fcbfdc38a3ff4605a605895f17169728" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 36225 } } peers { permanent_uuid: "09f190a507c34a28b35fc407add36bfd" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 33277 } }
I20260811 11:52:56.934721 25537 leader_election.cc:288] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5 [CANDIDATE]: Term 1 election: Requested vote from peers fcbfdc38a3ff4605a605895f17169728 (127.3.221.67:36225), 09f190a507c34a28b35fc407add36bfd (127.3.221.65:33277)
I20260811 11:52:56.935256 25405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d117b40d18fa4c32bfebb3491048c79d" candidate_uuid: "5b9318b4d8314adfa368a3a85228fca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcbfdc38a3ff4605a605895f17169728"
I20260811 11:52:56.935396 25405 raft_consensus.cc:3037] T d117b40d18fa4c32bfebb3491048c79d P fcbfdc38a3ff4605a605895f17169728 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:52:56.936551 25405 raft_consensus.cc:2454] T d117b40d18fa4c32bfebb3491048c79d P fcbfdc38a3ff4605a605895f17169728 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b9318b4d8314adfa368a3a85228fca5 in term 1.
I20260811 11:52:56.937147 25093 leader_election.cc:302] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b9318b4d8314adfa368a3a85228fca5, fcbfdc38a3ff4605a605895f17169728; no voters:
I20260811 11:52:56.937525 25537 raft_consensus.cc:2781] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:52:56.937587 24964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d117b40d18fa4c32bfebb3491048c79d" candidate_uuid: "5b9318b4d8314adfa368a3a85228fca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f190a507c34a28b35fc407add36bfd"
I20260811 11:52:56.937757 25537 raft_consensus.cc:686] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5 [term 1 LEADER]: Becoming Leader. State: Replica: 5b9318b4d8314adfa368a3a85228fca5, State: Running, Role: LEADER
I20260811 11:52:56.937726 24964 raft_consensus.cc:3037] T d117b40d18fa4c32bfebb3491048c79d P 09f190a507c34a28b35fc407add36bfd [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:52:56.938158 25537 consensus_queue.cc:237] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9318b4d8314adfa368a3a85228fca5" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 32805 } } peers { permanent_uuid: "fcbfdc38a3ff4605a605895f17169728" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 36225 } } peers { permanent_uuid: "09f190a507c34a28b35fc407add36bfd" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 33277 } }
I20260811 11:52:56.939138 24964 raft_consensus.cc:2454] T d117b40d18fa4c32bfebb3491048c79d P 09f190a507c34a28b35fc407add36bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b9318b4d8314adfa368a3a85228fca5 in term 1.
I20260811 11:52:56.942807 24759 catalog_manager.cc:5495] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b9318b4d8314adfa368a3a85228fca5 (127.3.221.66). New cstate: current_term: 1 leader_uuid: "5b9318b4d8314adfa368a3a85228fca5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9318b4d8314adfa368a3a85228fca5" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 32805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcbfdc38a3ff4605a605895f17169728" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 36225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f190a507c34a28b35fc407add36bfd" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 33277 } health_report { overall_health: UNKNOWN } } }
I20260811 11:52:56.997606 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:56.994012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36288 (local address 127.3.221.66:32805)
0811 11:52:56.994164 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:56.994169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:56.994183 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:52:56.994275 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:56.994278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:56.994335 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:56.994433 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:52:56.994689 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.995194 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.996051 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:56.996281 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:56.996373 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:56.996697 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:52:56.997165 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:52:56.997328 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:56.997361 (+ 33us) server_negotiation.cc:299] Negotiation successful
0811 11:52:56.997427 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 11:52:57.004544 25405 raft_consensus.cc:1261] T d117b40d18fa4c32bfebb3491048c79d P fcbfdc38a3ff4605a605895f17169728 [term 1 FOLLOWER]: Refusing update from remote peer 5b9318b4d8314adfa368a3a85228fca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:52:57.004544 24964 raft_consensus.cc:1261] T d117b40d18fa4c32bfebb3491048c79d P 09f190a507c34a28b35fc407add36bfd [term 1 FOLLOWER]: Refusing update from remote peer 5b9318b4d8314adfa368a3a85228fca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:52:57.005422 25537 consensus_queue.cc:1035] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f190a507c34a28b35fc407add36bfd" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 33277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:52:57.006022 25559 consensus_queue.cc:1035] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcbfdc38a3ff4605a605895f17169728" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 36225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:52:57.012804 25567 mvcc.cc:205] Tried to move back new op lower bound from 7317295829002149888 to 7317295828744372224. Current Snapshot: MvccSnapshot[applied={T|T < 7317295829002149888}]
I20260811 11:52:57.013784 25569 mvcc.cc:205] Tried to move back new op lower bound from 7317295829002149888 to 7317295828744372224. Current Snapshot: MvccSnapshot[applied={T|T < 7317295829002149888}]
I20260811 11:52:57.027763 24759 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:52:57.027966 24759 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:52:57.028313 25571 mvcc.cc:205] Tried to move back new op lower bound from 7317295829002149888 to 7317295828744372224. Current Snapshot: MvccSnapshot[applied={T|T < 7317295829002149888 or (T in {7317295829002149888})}]
I20260811 11:52:57.030834 24759 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4cc67b9dac514daeb5175042e57f5cc4: Sending DeleteTablet for 3 replicas of tablet d117b40d18fa4c32bfebb3491048c79d
I20260811 11:52:57.031675 25165 tablet_service.cc:1514] Processing DeleteTablet for tablet d117b40d18fa4c32bfebb3491048c79d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:52:57 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:36278
I20260811 11:52:57.031742 24960 tablet_service.cc:1514] Processing DeleteTablet for tablet d117b40d18fa4c32bfebb3491048c79d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:52:57 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:58204
I20260811 11:52:57.031982 25377 tablet_service.cc:1514] Processing DeleteTablet for tablet d117b40d18fa4c32bfebb3491048c79d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:52:57 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:60658
I20260811 11:52:57.032253 25585 tablet_replica.cc:330] T d117b40d18fa4c32bfebb3491048c79d P 09f190a507c34a28b35fc407add36bfd: stopping tablet replica
I20260811 11:52:57.032449 25586 tablet_replica.cc:330] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5: stopping tablet replica
I20260811 11:52:57.032665 25585 raft_consensus.cc:2229] T d117b40d18fa4c32bfebb3491048c79d P 09f190a507c34a28b35fc407add36bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:52:57.032807 25586 raft_consensus.cc:2229] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:52:57.033035 25585 raft_consensus.cc:2258] T d117b40d18fa4c32bfebb3491048c79d P 09f190a507c34a28b35fc407add36bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:52:57.034384 25586 raft_consensus.cc:2258] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:52:57.034796 25587 tablet_replica.cc:330] T d117b40d18fa4c32bfebb3491048c79d P fcbfdc38a3ff4605a605895f17169728: stopping tablet replica
I20260811 11:52:57.033088 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 24824
I20260811 11:52:57.035223 25587 raft_consensus.cc:2229] T d117b40d18fa4c32bfebb3491048c79d P fcbfdc38a3ff4605a605895f17169728 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:52:57.035609 25587 raft_consensus.cc:2258] T d117b40d18fa4c32bfebb3491048c79d P fcbfdc38a3ff4605a605895f17169728 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:52:57.035990 25586 ts_tablet_manager.cc:1905] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:52:57.038983 25586 ts_tablet_manager.cc:1918] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:52:57.039109 25586 log.cc:1196] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/d117b40d18fa4c32bfebb3491048c79d
I20260811 11:52:57.039285 25587 ts_tablet_manager.cc:1905] T d117b40d18fa4c32bfebb3491048c79d P fcbfdc38a3ff4605a605895f17169728: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:52:57.039480 25586 ts_tablet_manager.cc:1939] T d117b40d18fa4c32bfebb3491048c79d P 5b9318b4d8314adfa368a3a85228fca5: Deleting consensus metadata
I20260811 11:52:57.042246 25587 ts_tablet_manager.cc:1918] T d117b40d18fa4c32bfebb3491048c79d P fcbfdc38a3ff4605a605895f17169728: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:52:57.042456 25587 log.cc:1196] T d117b40d18fa4c32bfebb3491048c79d P fcbfdc38a3ff4605a605895f17169728: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/d117b40d18fa4c32bfebb3491048c79d
I20260811 11:52:57.042861 25587 ts_tablet_manager.cc:1939] T d117b40d18fa4c32bfebb3491048c79d P fcbfdc38a3ff4605a605895f17169728: Deleting consensus metadata
I20260811 11:52:57.044061 24739 catalog_manager.cc:4841] TS 5b9318b4d8314adfa368a3a85228fca5 (127.3.221.66:32805): tablet d117b40d18fa4c32bfebb3491048c79d (table test-table [id=6f39cd54ee2f4a2baa12a6860ad85d7f]) successfully deleted
I20260811 11:52:57.044464 24737 catalog_manager.cc:4841] TS fcbfdc38a3ff4605a605895f17169728 (127.3.221.67:36225): tablet d117b40d18fa4c32bfebb3491048c79d (table test-table [id=6f39cd54ee2f4a2baa12a6860ad85d7f]) successfully deleted
W20260811 11:52:57.047941 24737 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.221.65:33277 (error 108)
I20260811 11:52:57.047992 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 25062
I20260811 11:52:57.049405 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:57.049033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:33277 (local address 127.0.0.1:58220)
0811 11:52:57.049247 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:57.049309 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.65:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260811 11:52:57.049679 24737 catalog_manager.cc:4584] TS 09f190a507c34a28b35fc407add36bfd (127.3.221.65:33277): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d117b40d18fa4c32bfebb3491048c79d: Network error: Client connection negotiation failed: client connection to 127.3.221.65:33277: connect: Connection refused (error 111)
I20260811 11:52:57.064221 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 25276
I20260811 11:52:57.087491 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 24709
I20260811 11:52:57.109875 3957 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260811 11:52:57.457968 3957 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260811 11:52:57.458101 3957 mini_ranger.cc:66] Stopping Ranger...
I20260811 11:52:57.804641 3957 mini_ranger.cc:68] Stopped Ranger
2026-08-11 11:52:57.804 UTC [23310] LOG: received smart shutdown request
2026-08-11 11:52:57.809 UTC [23310] LOG: background worker "logical replication launcher" (PID 23325) exited with exit code 1
2026-08-11 11:52:57.809 UTC [23315] LOG: shutting down
2026-08-11 11:52:57.999 UTC [23310] LOG: database system is shut down
2026-08-11T11:52:58Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53655 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:52:58 dist-test-slave-bndk krb5kdc[25600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:52:58 dist-test-slave-bndk krb5kdc[25600](info): set up 2 sockets
Aug 11 11:52:58 dist-test-slave-bndk krb5kdc[25600](info): commencing operation
krb5kdc: starting...
W20260811 11:52:59.296622 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:52:59 dist-test-slave-bndk krb5kdc[25600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:52:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:52:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:52:59.439750 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:38717
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:38717,127.3.221.125:39703,127.3.221.124:33769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:52:59.562608 25620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:52:59.562933 25620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:52:59.563007 25620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:52:59.568387 25620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:52:59.568519 25620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:52:59.568557 25620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:52:59.568580 25620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:52:59.568600 25620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:52:59.576339 25620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:38717,127.3.221.125:39703,127.3.221.124:33769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:38717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:52:59.577769 25620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:52:59.578929 25620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:52:59.588515 25630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:52:59.588586 25632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:52:59.588557 25629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:52:59.589169 25620 server_base.cc:1034] running on GCE node
I20260811 11:52:59.589870 25620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:52:59.591138 25620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:52:59.592342 25620 hybrid_clock.cc:648] HybridClock initialized: now 1786449179592319 us; error 44 us; skew 500 ppm
Aug 11 11:52:59 dist-test-slave-bndk krb5kdc[25600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449179, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:52:59.595587 25620 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:52:59.596728 25620 webserver.cc:460] Webserver started at http://127.3.221.126:40603/ using document root <none> and password file <none>
I20260811 11:52:59.597425 25620 fs_manager.cc:359] Metadata directory not provided
I20260811 11:52:59.597507 25620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:52:59.597728 25620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:52:59.599977 25620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "be1124c123494619a6294d6f0db4e948"
format_stamp: "Formatted at 2026-08-11 11:52:59 on dist-test-slave-bndk"
server_key: "f1e738d58e1a1fb49fd073a6df46c266"
server_key_iv: "b4c516823ec8e597cf3eca56ebf25098"
server_key_version: "encryptionkey@0"
I20260811 11:52:59.600602 25620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "be1124c123494619a6294d6f0db4e948"
format_stamp: "Formatted at 2026-08-11 11:52:59 on dist-test-slave-bndk"
server_key: "f1e738d58e1a1fb49fd073a6df46c266"
server_key_iv: "b4c516823ec8e597cf3eca56ebf25098"
server_key_version: "encryptionkey@0"
I20260811 11:52:59.605062 25620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 11:52:59.607914 25645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:59.609452 25620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:52:59.609601 25620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "be1124c123494619a6294d6f0db4e948"
format_stamp: "Formatted at 2026-08-11 11:52:59 on dist-test-slave-bndk"
server_key: "f1e738d58e1a1fb49fd073a6df46c266"
server_key_iv: "b4c516823ec8e597cf3eca56ebf25098"
server_key_version: "encryptionkey@0"
I20260811 11:52:59.609730 25620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:52:59.622596 25620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:52:59.626506 25620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:52:59.626739 25620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:52:59.634527 25620 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:38717
I20260811 11:52:59.634552 25729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:38717 every 8 connection(s)
I20260811 11:52:59.635661 25620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:52:59.638569 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 25620
I20260811 11:52:59.638715 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:52:59.638988 25730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:52:59.639060 3957 external_mini_cluster.cc:1428] Setting key dbcd12ffa430359eb5fa598cf56ce84c
I20260811 11:52:59.644001 25730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38717 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.221.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.125@KRBTEST.COM" created.
Aug 11 11:52:59 dist-test-slave-bndk krb5kdc[25600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449179, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:52:59.660408 25734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:59.646682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43332 (local address 127.3.221.126:38717)
0811 11:52:59.647135 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:59.647156 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:59.647500 (+ 344us) server_negotiation.cc:407] Connection header received
0811 11:52:59.648854 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:59.648859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:59.649060 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:59.649262 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:52:59.650171 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:59.651061 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:59.652091 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:59.652499 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:59.652655 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:59.655548 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:52:59.655553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:52:59.655565 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:52:59.655589 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:52:59.657155 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:59.657647 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:59.657652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:59.657653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:59.657773 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:59.658063 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:59.658067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:59.658068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:59.658489 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:52:59.658617 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:59.658988 (+ 371us) server_negotiation.cc:299] Negotiation successful
0811 11:52:59.659233 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"spinlock_wait_cycles":4864,"thread_start_us":128,"threads_started":1}
I20260811 11:52:59.660403 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:59.646682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:38717 (local address 127.0.0.1:43332)
0811 11:52:59.647271 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:59.647309 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:59.648397 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:59.649299 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:52:59.649305 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:52:59.649748 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:52:59.650020 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:59.650036 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:59.651184 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:59.651187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:59.651981 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:59.651992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:59.652656 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:59.652660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:59.652755 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:59.653355 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:52:59.653373 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:52:59.655383 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:52:59.657283 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:59.657293 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:59.657304 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:59.657540 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:59.657859 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:59.657862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:59.657864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:59.657980 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:59.658600 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:52:59.658607 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:52:59.658749 (+ 142us) client_negotiation.cc:769] Sending connection context
0811 11:52:59.658908 (+ 159us) client_negotiation.cc:240] Negotiation successful
0811 11:52:59.659314 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":92,"spinlock_wait_cycles":1920,"thread_start_us":189,"threads_started":1}
I20260811 11:52:59.663142 25730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:52:59.663933 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:59.663656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:39703 (local address 127.0.0.1:43608)
0811 11:52:59.663780 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:59.663860 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260811 11:52:59.664152 25652 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.221.125:39703: connect: Connection refused (error 111)
I20260811 11:52:59.665110 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:59.664866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:39703 (local address 127.0.0.1:43624)
0811 11:52:59.664996 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:59.665047 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260811 11:52:59.665357 25730 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.125:39703: Network error: Client connection negotiation failed: client connection to 127.3.221.125:39703: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.125.keytab.
Entry for principal kudu/127.3.221.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.125.keytab.
I20260811 11:52:59.668602 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.125:39703
--webserver_interface=127.3.221.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.125.keytab
--principal=kudu/127.3.221.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:38717,127.3.221.125:39703,127.3.221.124:33769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:52:59.714608 25730 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } attempt: 1
I20260811 11:52:59.715545 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:59.715216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:39703 (local address 127.0.0.1:43634)
0811 11:52:59.715372 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:59.715450 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260811 11:52:59.716426 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:59.716115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:39703 (local address 127.0.0.1:43644)
0811 11:52:59.716281 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:59.716358 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260811 11:52:59.716563 25730 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.125:39703: Network error: Client connection negotiation failed: client connection to 127.3.221.125:39703: connect: Connection refused (error 111)
I20260811 11:52:59.784734 25730 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } attempt: 2
I20260811 11:52:59.785694 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:59.785369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:39703 (local address 127.0.0.1:43660)
0811 11:52:59.785520 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:59.785610 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260811 11:52:59.786602 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:59.786345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:39703 (local address 127.0.0.1:43662)
0811 11:52:59.786483 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:59.786534 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260811 11:52:59.786738 25730 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.125:39703: Network error: Client connection negotiation failed: client connection to 127.3.221.125:39703: connect: Connection refused (error 111)
W20260811 11:52:59.802685 25739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:52:59.803010 25739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:52:59.803052 25739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:52:59.807283 25739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:52:59.807372 25739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:52:59.807391 25739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:52:59.807407 25739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:52:59.807422 25739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:52:59.813210 25739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:38717,127.3.221.125:39703,127.3.221.124:33769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.125.keytab
--principal=kudu/127.3.221.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.125:39703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.221.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:52:59.814518 25739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:52:59.815626 25739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:52:59.824970 25749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:52:59.825060 25746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:52:59.825212 25739 server_base.cc:1034] running on GCE node
W20260811 11:52:59.824970 25747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:52:59.826125 25739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:52:59.827297 25739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:52:59.828491 25739 hybrid_clock.cc:648] HybridClock initialized: now 1786449179828462 us; error 60 us; skew 500 ppm
Aug 11 11:52:59 dist-test-slave-bndk krb5kdc[25600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449179, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:52:59.831845 25739 init.cc:377] Logged in from keytab as kudu/127.3.221.125@KRBTEST.COM (short username kudu)
I20260811 11:52:59.833061 25739 webserver.cc:460] Webserver started at http://127.3.221.125:41667/ using document root <none> and password file <none>
I20260811 11:52:59.833719 25739 fs_manager.cc:359] Metadata directory not provided
I20260811 11:52:59.833794 25739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:52:59.834007 25739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:52:59.836365 25739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-1/data/instance:
uuid: "d7952c86f2cc4827ac9e3b6940ec9054"
format_stamp: "Formatted at 2026-08-11 11:52:59 on dist-test-slave-bndk"
server_key: "c7c1f7c36cedf92158e43a8bc82ac780"
server_key_iv: "b6d921f98b4d0e7eff8301914249563e"
server_key_version: "encryptionkey@0"
I20260811 11:52:59.836989 25739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-1/wal/instance:
uuid: "d7952c86f2cc4827ac9e3b6940ec9054"
format_stamp: "Formatted at 2026-08-11 11:52:59 on dist-test-slave-bndk"
server_key: "c7c1f7c36cedf92158e43a8bc82ac780"
server_key_iv: "b6d921f98b4d0e7eff8301914249563e"
server_key_version: "encryptionkey@0"
I20260811 11:52:59.841164 25739 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 11:52:59.844166 25762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:52:59.845860 25739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:52:59.846004 25739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-1/wal
uuid: "d7952c86f2cc4827ac9e3b6940ec9054"
format_stamp: "Formatted at 2026-08-11 11:52:59 on dist-test-slave-bndk"
server_key: "c7c1f7c36cedf92158e43a8bc82ac780"
server_key_iv: "b6d921f98b4d0e7eff8301914249563e"
server_key_version: "encryptionkey@0"
I20260811 11:52:59.846118 25739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:52:59.877935 25730 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } attempt: 3
I20260811 11:52:59.878844 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:59.878529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:39703 (local address 127.0.0.1:43666)
0811 11:52:59.878688 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:59.878749 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260811 11:52:59.879526 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:59.879315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:39703 (local address 127.0.0.1:43680)
0811 11:52:59.879420 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:59.879465 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260811 11:52:59.879667 25730 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.125:39703: Network error: Client connection negotiation failed: client connection to 127.3.221.125:39703: connect: Connection refused (error 111)
I20260811 11:52:59.881002 25739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:52:59.884856 25739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:52:59.885059 25739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:52:59.892464 25739 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.125:39703
I20260811 11:52:59.892513 25850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.125:39703 every 8 connection(s)
I20260811 11:52:59.893534 25739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-1/data/info.pb
I20260811 11:52:59.896531 25851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:52:59.899811 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 25739
I20260811 11:52:59.899986 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-1/wal/instance
I20260811 11:52:59.900354 3957 external_mini_cluster.cc:1428] Setting key edebdde946c7d30b72ce10a1e200edaa
I20260811 11:52:59.900730 25851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38717 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.221.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.124@KRBTEST.COM" created.
Aug 11 11:52:59 dist-test-slave-bndk krb5kdc[25600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449179, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.125@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:52:59.921657 25734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:59.903179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43334 (local address 127.3.221.126:38717)
0811 11:52:59.903358 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:59.903363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:59.904343 (+ 980us) server_negotiation.cc:407] Connection header received
0811 11:52:59.905398 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:59.905402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:59.905473 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:59.905558 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:52:59.907603 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:59.908528 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:59.909957 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:59.910454 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:59.910609 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:59.916784 (+ 6175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:52:59.916789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:52:59.916792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:52:59.916847 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:52:59.918914 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:59.919647 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:59.919651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:59.919653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:59.919764 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:59.920143 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:59.920146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:59.920148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:59.920430 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:52:59.920561 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:59.921302 (+ 741us) server_negotiation.cc:299] Negotiation successful
0811 11:52:59.921466 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 11:52:59.922469 25854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:59.903636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:38717 (local address 127.0.0.1:43334)
0811 11:52:59.904161 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:59.904219 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:59.905117 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:59.905971 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:52:59.905984 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:52:59.906503 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:52:59.907349 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:59.907373 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:59.908764 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:59.908768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:59.909793 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:59.909806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:59.910682 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:59.910686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:59.910857 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:59.911562 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:52:59.911580 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:52:59.916433 (+ 4853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:52:59.919111 (+ 2678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:59.919118 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:59.919136 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:59.919494 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:59.919874 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:59.919878 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:59.919880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:59.920042 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:59.920578 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:52:59.920585 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:52:59.920958 (+ 373us) client_negotiation.cc:769] Sending connection context
0811 11:52:59.921143 (+ 185us) client_negotiation.cc:240] Negotiation successful
0811 11:52:59.921527 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":141,"threads_started":1}
I20260811 11:52:59.924170 25851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:52:59 dist-test-slave-bndk krb5kdc[25600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449179, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.125@KRBTEST.COM for kudu/127.3.221.125@KRBTEST.COM
I20260811 11:52:59.935202 25854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:59.924764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:39703 (local address 127.0.0.1:43684)
0811 11:52:59.924979 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:59.924994 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:52:59.925083 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:52:59.925726 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:52:59.925729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:52:59.925878 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:52:59.926006 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:59.926013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:59.927132 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:59.927135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:59.927735 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:52:59.927745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:52:59.928432 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:52:59.928435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:52:59.928523 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:59.929006 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:52:59.929018 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:52:59.930540 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:52:59.933058 (+ 2518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:59.933062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:59.933064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:59.933375 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:59.934013 (+ 638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:52:59.934015 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:52:59.934017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:52:59.934112 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:52:59.934733 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:52:59.934735 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:52:59.934828 (+ 93us) client_negotiation.cc:769] Sending connection context
0811 11:52:59.934920 (+ 92us) client_negotiation.cc:240] Negotiation successful
0811 11:52:59.935056 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260811 11:52:59.935359 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:59.924900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43684 (local address 127.3.221.125:39703)
0811 11:52:59.925317 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0811 11:52:59.925330 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 11:52:59.925355 (+ 25us) server_negotiation.cc:407] Connection header received
0811 11:52:59.925403 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:52:59.925409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:52:59.925578 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:52:59.925692 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:52:59.926153 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:59.927004 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:59.927877 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:52:59.928271 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:52:59.928382 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:52:59.930825 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:52:59.930845 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:52:59.930874 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:52:59.930921 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:52:59.932915 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:59.933604 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:59.933617 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:59.933619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:59.933838 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:52:59.934240 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:52:59.934245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:52:59.934247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:52:59.934600 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:52:59.934736 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0811 11:52:59.935003 (+ 267us) server_negotiation.cc:299] Negotiation successful
0811 11:52:59.935163 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":172,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.124.keytab.
Entry for principal kudu/127.3.221.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.124.keytab.
I20260811 11:52:59.937376 25851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:52:59.937692 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.124:33769
--webserver_interface=127.3.221.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.124.keytab
--principal=kudu/127.3.221.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:38717,127.3.221.125:39703,127.3.221.124:33769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:52:59.938175 25854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:59.937871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:33769 (local address 127.0.0.1:43430)
0811 11:52:59.938015 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:59.938099 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260811 11:52:59.938434 25766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.221.124:33769: connect: Connection refused (error 111)
I20260811 11:52:59.939368 25854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:59.939130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:33769 (local address 127.0.0.1:43432)
0811 11:52:59.939245 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:59.939301 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260811 11:52:59.939527 25851 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.124:33769: Network error: Client connection negotiation failed: client connection to 127.3.221.124:33769: connect: Connection refused (error 111)
I20260811 11:52:59.988811 25851 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } attempt: 1
I20260811 11:52:59.990087 25854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:59.989498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:33769 (local address 127.0.0.1:43448)
0811 11:52:59.989836 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:59.989951 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260811 11:52:59.991161 25854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:52:59.990825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:33769 (local address 127.0.0.1:43462)
0811 11:52:59.991017 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0811 11:52:59.991082 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260811 11:52:59.991415 25851 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.124:33769: Network error: Client connection negotiation failed: client connection to 127.3.221.124:33769: connect: Connection refused (error 111)
I20260811 11:53:00.022843 25730 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } attempt: 4
Aug 11 11:53:00 dist-test-slave-bndk krb5kdc[25600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449179, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for kudu/127.3.221.125@KRBTEST.COM
I20260811 11:53:00.033655 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.023468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:39703 (local address 127.0.0.1:43686)
0811 11:53:00.023816 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:00.023838 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:00.024018 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:00.024376 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:00.024380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:00.024695 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:53:00.025410 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:00.025427 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.026923 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.026926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:00.027545 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:00.027555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.028068 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.028071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:00.028180 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:00.028681 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:53:00.028694 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:53:00.030317 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:53:00.032083 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:00.032086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:00.032089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:00.032300 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:00.032562 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:00.032566 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:00.032568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:00.032652 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:00.033044 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:53:00.033048 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:53:00.033154 (+ 106us) client_negotiation.cc:769] Sending connection context
0811 11:53:00.033284 (+ 130us) client_negotiation.cc:240] Negotiation successful
0811 11:53:00.033460 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260811 11:53:00.033655 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.023780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43686 (local address 127.3.221.125:39703)
0811 11:53:00.024005 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:00.024010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:00.024038 (+ 28us) server_negotiation.cc:407] Connection header received
0811 11:53:00.024179 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:00.024182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:00.024242 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:00.024321 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:00.025611 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.026748 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.027676 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.027945 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.028043 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:00.030494 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:00.030497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:00.030500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:00.030528 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:00.031962 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.032397 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.032400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.032401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.032477 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.032735 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.032738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.032740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.032948 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:00.033038 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:00.033356 (+ 318us) server_negotiation.cc:299] Negotiation successful
0811 11:53:00.033478 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 11:53:00.034719 25730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:53:00.035465 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.035213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:33769 (local address 127.0.0.1:43476)
0811 11:53:00.035341 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:00.035393 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260811 11:53:00.036074 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.035866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:33769 (local address 127.0.0.1:43484)
0811 11:53:00.035969 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:00.036017 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260811 11:53:00.036181 25730 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.124:33769: Network error: Client connection negotiation failed: client connection to 127.3.221.124:33769: connect: Connection refused (error 111)
I20260811 11:53:00.059625 25851 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } attempt: 2
I20260811 11:53:00.060667 25854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.060215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:33769 (local address 127.0.0.1:43492)
0811 11:53:00.060478 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:00.060564 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260811 11:53:00.062449 25854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.062108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:33769 (local address 127.0.0.1:43494)
0811 11:53:00.062285 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:00.062362 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260811 11:53:00.062718 25851 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.124:33769: Network error: Client connection negotiation failed: client connection to 127.3.221.124:33769: connect: Connection refused (error 111)
W20260811 11:53:00.070101 25858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:53:00.070423 25858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:53:00.070658 25858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:53:00.076419 25858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:53:00.076664 25858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:53:00.076771 25858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:53:00.076809 25858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:53:00.076843 25858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:53:00.084407 25858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:38717,127.3.221.125:39703,127.3.221.124:33769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.124.keytab
--principal=kudu/127.3.221.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.124:33769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.221.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:53:00.085776 25858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:53:00.086923 25858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:53:00.095340 25730 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } attempt: 1
I20260811 11:53:00.096392 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.096031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:33769 (local address 127.0.0.1:43504)
0811 11:53:00.096179 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:00.096296 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260811 11:53:00.097111 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.096894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:33769 (local address 127.0.0.1:43506)
0811 11:53:00.096996 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:00.097049 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260811 11:53:00.097196 25869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:00.097452 25730 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.124:33769: Network error: Client connection negotiation failed: client connection to 127.3.221.124:33769: connect: Connection refused (error 111)
W20260811 11:53:00.097551 25867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:00.097740 25873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:00.097890 25858 server_base.cc:1034] running on GCE node
I20260811 11:53:00.098820 25858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:53:00.100756 25858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:53:00.102043 25858 hybrid_clock.cc:648] HybridClock initialized: now 1786449180102016 us; error 53 us; skew 500 ppm
Aug 11 11:53:00 dist-test-slave-bndk krb5kdc[25600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449180, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:53:00.105434 25858 init.cc:377] Logged in from keytab as kudu/127.3.221.124@KRBTEST.COM (short username kudu)
I20260811 11:53:00.106803 25858 webserver.cc:460] Webserver started at http://127.3.221.124:41197/ using document root <none> and password file <none>
I20260811 11:53:00.107482 25858 fs_manager.cc:359] Metadata directory not provided
I20260811 11:53:00.107565 25858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:53:00.107796 25858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:53:00.110334 25858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-2/data/instance:
uuid: "395fd6ac31db4c4aa65b945125d12e2d"
format_stamp: "Formatted at 2026-08-11 11:53:00 on dist-test-slave-bndk"
server_key: "f1551e7899cc90ddacd35a250c227360"
server_key_iv: "9fa7caf2f5572f94ff51e6b63cb0cbc8"
server_key_version: "encryptionkey@0"
I20260811 11:53:00.111012 25858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-2/wal/instance:
uuid: "395fd6ac31db4c4aa65b945125d12e2d"
format_stamp: "Formatted at 2026-08-11 11:53:00 on dist-test-slave-bndk"
server_key: "f1551e7899cc90ddacd35a250c227360"
server_key_iv: "9fa7caf2f5572f94ff51e6b63cb0cbc8"
server_key_version: "encryptionkey@0"
I20260811 11:53:00.115955 25858 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 11:53:00.120496 25891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:00.122593 25858 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 11:53:00.122900 25858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-2/wal
uuid: "395fd6ac31db4c4aa65b945125d12e2d"
format_stamp: "Formatted at 2026-08-11 11:53:00 on dist-test-slave-bndk"
server_key: "f1551e7899cc90ddacd35a250c227360"
server_key_iv: "9fa7caf2f5572f94ff51e6b63cb0cbc8"
server_key_version: "encryptionkey@0"
I20260811 11:53:00.123072 25858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:53:00.153966 25851 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } attempt: 3
I20260811 11:53:00.154938 25854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.154602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:33769 (local address 127.0.0.1:43520)
0811 11:53:00.154784 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:00.154850 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260811 11:53:00.156188 25854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.155621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:33769 (local address 127.0.0.1:43534)
0811 11:53:00.155955 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:00.156053 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":112}
W20260811 11:53:00.156435 25851 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.124:33769: Network error: Client connection negotiation failed: client connection to 127.3.221.124:33769: connect: Connection refused (error 111)
I20260811 11:53:00.158833 25858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:53:00.162397 25858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:53:00.162614 25858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:53:00.164688 25730 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } attempt: 2
I20260811 11:53:00.165848 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.165452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:33769 (local address 127.0.0.1:43542)
0811 11:53:00.165627 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:00.165709 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260811 11:53:00.167055 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.166540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:33769 (local address 127.0.0.1:43554)
0811 11:53:00.166820 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:00.166932 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260811 11:53:00.167419 25730 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.124:33769: Network error: Client connection negotiation failed: client connection to 127.3.221.124:33769: connect: Connection refused (error 111)
I20260811 11:53:00.170475 25858 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.124:33769
I20260811 11:53:00.170698 25972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.124:33769 every 8 connection(s)
I20260811 11:53:00.171689 25858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-2/data/info.pb
I20260811 11:53:00.174944 25973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:53:00.177735 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 25858
I20260811 11:53:00.177893 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786449004114841-3957-0/minicluster-data/master-2/wal/instance
I20260811 11:53:00.178231 3957 external_mini_cluster.cc:1428] Setting key db7f3452b3e6baf786f9700f2608594a
I20260811 11:53:00.179834 25973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38717 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:53:00 dist-test-slave-bndk krb5kdc[25600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449180, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.124@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
Aug 11 11:53:00 dist-test-slave-bndk krb5kdc[25600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.124@KRBTEST.COM
I20260811 11:53:00.193305 25734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.181920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43350 (local address 127.3.221.126:38717)
0811 11:53:00.182272 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:00.182280 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:00.182324 (+ 44us) server_negotiation.cc:407] Connection header received
0811 11:53:00.182481 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:00.182485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:00.182559 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:00.182701 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:00.183826 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.184635 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.185886 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.186120 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.186218 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:00.189042 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:00.189045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:00.189048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:00.189076 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:00.190579 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.191151 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.191154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.191155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.191262 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.191720 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.191727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.191729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.192136 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:00.192284 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:00.192965 (+ 681us) server_negotiation.cc:299] Negotiation successful
0811 11:53:00.193115 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 11:53:00.194416 25980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.179764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43566 (local address 127.3.221.124:33769)
0811 11:53:00.180255 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:00.180280 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:00.180331 (+ 51us) server_negotiation.cc:407] Connection header received
0811 11:53:00.181045 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:00.181065 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:00.181514 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:00.181954 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:00.183453 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.184882 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.185863 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.186286 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.186433 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:00.189563 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:00.189572 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:00.189590 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:00.189613 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:00.191342 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.192170 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.192175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.192177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.192281 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.192842 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.192846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.192848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.193149 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:00.193412 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:00.193665 (+ 253us) server_negotiation.cc:299] Negotiation successful
0811 11:53:00.193814 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":329,"thread_start_us":142,"threads_started":1}
I20260811 11:53:00.194419 25986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.181750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:38717 (local address 127.0.0.1:43350)
0811 11:53:00.182163 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:00.182218 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:00.182325 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:00.182790 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:00.182797 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:00.183278 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:53:00.183638 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:00.183650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.184786 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.184788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:00.185759 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:00.185771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.186276 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.186280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:00.186424 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:00.187126 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:53:00.187143 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:53:00.188891 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:53:00.190692 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:00.190701 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:00.190715 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:00.191032 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:00.191378 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:00.191381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:00.191383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:00.191555 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:00.192308 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:53:00.192316 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:53:00.192666 (+ 350us) client_negotiation.cc:769] Sending connection context
0811 11:53:00.192872 (+ 206us) client_negotiation.cc:240] Negotiation successful
0811 11:53:00.193271 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":52,"spinlock_wait_cycles":7040,"thread_start_us":132,"threads_started":1}
I20260811 11:53:00.195849 25973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:53:00 dist-test-slave-bndk krb5kdc[25600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449180, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.124@KRBTEST.COM for kudu/127.3.221.125@KRBTEST.COM
Aug 11 11:53:00 dist-test-slave-bndk krb5kdc[25600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:53:00.204811 25986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.196362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:39703 (local address 127.0.0.1:43696)
0811 11:53:00.196490 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:00.196503 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:00.196577 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:00.196836 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:00.196839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:00.197012 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:53:00.197136 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:00.197143 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.197950 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.197952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:00.198613 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:00.198625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.199094 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.199098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:00.199204 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:00.199672 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:53:00.199684 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:53:00.201182 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:53:00.203058 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:00.203062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:00.203064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:00.203330 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:00.203677 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:00.203681 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:00.203683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:00.203778 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:00.204263 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:53:00.204267 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:53:00.204397 (+ 130us) client_negotiation.cc:769] Sending connection context
0811 11:53:00.204504 (+ 107us) client_negotiation.cc:240] Negotiation successful
0811 11:53:00.204637 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260811 11:53:00.204890 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.196504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43696 (local address 127.3.221.125:39703)
0811 11:53:00.196646 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:00.196651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:00.196665 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:53:00.196702 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:00.196704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:00.196745 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:00.196809 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:00.197304 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.197830 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.198765 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.198983 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.199078 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:00.201353 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:00.201356 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:00.201358 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:00.201386 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:00.202942 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.203485 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.203488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.203489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.203580 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.203881 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.203884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.203886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.204141 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:00.204257 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:00.204610 (+ 353us) server_negotiation.cc:299] Negotiation successful
0811 11:53:00.204745 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 11:53:00.205631 25973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:53:00.205950 25734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.197212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43360 (local address 127.3.221.126:38717)
0811 11:53:00.197382 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:00.197387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:00.197401 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:53:00.197442 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:00.197445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:00.197490 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:00.197561 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:00.198183 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.198668 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.199404 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.199625 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.199709 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:00.202165 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:00.202168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:00.202170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:00.202196 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:00.203754 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.204263 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.204267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.204268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.204378 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.204723 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.204727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.204729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.204976 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:00.205240 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:00.205598 (+ 358us) server_negotiation.cc:299] Negotiation successful
0811 11:53:00.205726 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Aug 11 11:53:00 dist-test-slave-bndk krb5kdc[25600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449180, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.124@KRBTEST.COM for kudu/127.3.221.124@KRBTEST.COM
Aug 11 11:53:00 dist-test-slave-bndk krb5kdc[25600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.125@KRBTEST.COM
I20260811 11:53:00.215137 25986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.206131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:33769 (local address 127.0.0.1:43578)
0811 11:53:00.206245 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:00.206257 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:00.206333 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:00.206684 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:00.206687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:00.206841 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:53:00.206926 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:00.206933 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.208419 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.208422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:00.209245 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:00.209256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.210027 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.210030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:00.210102 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:00.210596 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:53:00.210610 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:53:00.211950 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:53:00.213729 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:00.213732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:00.213735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:00.213914 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:00.214152 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:00.214155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:00.214158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:00.214228 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:00.214614 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:53:00.214617 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:53:00.214708 (+ 91us) client_negotiation.cc:769] Sending connection context
0811 11:53:00.214791 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 11:53:00.214945 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260811 11:53:00.215137 25980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.206392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43578 (local address 127.3.221.124:33769)
0811 11:53:00.206478 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:00.206481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:00.206491 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:53:00.206519 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:00.206521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:00.206553 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:00.206644 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:00.207081 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.208290 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.209393 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.209877 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.210004 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:00.212116 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:00.212119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:00.212121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:00.212144 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:00.213634 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.213999 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.214003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.214004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.214074 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.214305 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.214308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.214310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.214525 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:00.214615 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:00.214847 (+ 232us) server_negotiation.cc:299] Negotiation successful
0811 11:53:00.215000 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":34}
I20260811 11:53:00.216966 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.207334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43704 (local address 127.3.221.125:39703)
0811 11:53:00.207434 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:00.207437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:00.207447 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:53:00.207499 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:00.207501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:00.207534 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:00.207598 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:00.208311 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.209092 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.210492 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.210717 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.210827 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:00.213591 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:00.213594 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:00.213596 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:00.213624 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:00.214992 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.215743 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.215746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.215747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.215836 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.216121 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.216124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.216126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.216345 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:00.216530 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:00.216717 (+ 187us) server_negotiation.cc:299] Negotiation successful
0811 11:53:00.216816 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260811 11:53:00.219058 25973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d: Bootstrap starting.
I20260811 11:53:00.220844 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:53:00.221822 25973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d: Neither blocks nor log segments found. Creating new log.
I20260811 11:53:00.222666 25973 log.cc:824] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d: Log is configured to *not* fsync() on all Append() calls
I20260811 11:53:00.225718 25973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d: No bootstrap required, opened a new log
I20260811 11:53:00.233809 25973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1124c123494619a6294d6f0db4e948" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38717 } } peers { permanent_uuid: "d7952c86f2cc4827ac9e3b6940ec9054" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } } peers { permanent_uuid: "395fd6ac31db4c4aa65b945125d12e2d" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } }
I20260811 11:53:00.234189 25973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:53:00.234251 25973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395fd6ac31db4c4aa65b945125d12e2d, State: Initialized, Role: FOLLOWER
I20260811 11:53:00.234953 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.223093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43714 (local address 127.3.221.125:39703)
0811 11:53:00.223214 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:00.223218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:00.223749 (+ 531us) server_negotiation.cc:407] Connection header received
0811 11:53:00.223800 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:00.223803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:00.223853 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:00.223935 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:00.224828 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.225448 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.226613 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.226944 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.227039 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:00.229135 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:00.229138 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:00.229141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:00.229168 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:00.231684 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.232238 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.232266 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.232268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.233271 (+ 1003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.233611 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.233615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.233616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.233865 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:00.234008 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:00.234646 (+ 638us) server_negotiation.cc:299] Negotiation successful
0811 11:53:00.234782 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 11:53:00.234889 25973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1124c123494619a6294d6f0db4e948" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38717 } } peers { permanent_uuid: "d7952c86f2cc4827ac9e3b6940ec9054" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } } peers { permanent_uuid: "395fd6ac31db4c4aa65b945125d12e2d" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } }
I20260811 11:53:00.236052 25980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.223729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43594 (local address 127.3.221.124:33769)
0811 11:53:00.223847 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:00.223851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:00.226229 (+ 2378us) server_negotiation.cc:407] Connection header received
0811 11:53:00.226472 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:00.226475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:00.226634 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:00.226727 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:00.228166 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.228935 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.230304 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.230650 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.230742 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:00.232403 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:00.232407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:00.232459 (+ 52us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:00.232499 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:00.234077 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.234520 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.234523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.234524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.234616 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.235050 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.235054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.235056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.235326 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:00.235426 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:00.235764 (+ 338us) server_negotiation.cc:299] Negotiation successful
0811 11:53:00.235905 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 11:53:00.236389 25734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.223151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43374 (local address 127.3.221.126:38717)
0811 11:53:00.223281 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:00.223285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:00.223546 (+ 261us) server_negotiation.cc:407] Connection header received
0811 11:53:00.225554 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:00.225557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:00.225619 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:00.225695 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:00.226617 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.227315 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.228533 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.228879 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.228978 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:00.231337 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:00.231341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:00.231344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:00.231376 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:00.233058 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.233814 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.233818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.233819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.233915 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.234531 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.234535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.234537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.234763 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:00.234877 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:00.235871 (+ 994us) server_negotiation.cc:299] Negotiation successful
0811 11:53:00.236002 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 11:53:00.236603 25973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:53:00.237301 26000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1124c123494619a6294d6f0db4e948" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38717 } } peers { permanent_uuid: "d7952c86f2cc4827ac9e3b6940ec9054" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } } peers { permanent_uuid: "395fd6ac31db4c4aa65b945125d12e2d" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } } }
I20260811 11:53:00.237589 26000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d [sys.catalog]: This master's current role is: FOLLOWER
W20260811 11:53:00.242369 26022 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:53:00.242462 26022 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:53:00.240317 25858 master_runner.cc:186] Error getting master registration for 127.3.221.126:38717: OK, instance_id { permanent_uuid: "be1124c123494619a6294d6f0db4e948" instance_seqno: 1786449179632124 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 11:53:00.243677 25858 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 11:53:00.267586 25730 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } attempt: 3
Aug 11 11:53:00 dist-test-slave-bndk krb5kdc[25600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449179, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for kudu/127.3.221.124@KRBTEST.COM
I20260811 11:53:00.278213 25980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.268314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43604 (local address 127.3.221.124:33769)
0811 11:53:00.268459 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:00.268463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:00.268474 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:53:00.268507 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:00.268509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:00.268545 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:00.268612 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:00.269064 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.269609 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.270359 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.270590 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.270681 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:00.273206 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:00.273209 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:00.273211 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:00.273272 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:00.275383 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.276780 (+ 1397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.276785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.276787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.276923 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.277302 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.277306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.277308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.277525 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:00.277628 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:00.277901 (+ 273us) server_negotiation.cc:299] Negotiation successful
0811 11:53:00.278038 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 11:53:00.278213 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.268037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:33769 (local address 127.0.0.1:43604)
0811 11:53:00.268211 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:00.268225 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:00.268314 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:00.268650 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:00.268653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:00.268852 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:53:00.268953 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:00.268959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.269711 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.269714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:00.270237 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:00.270246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.270696 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.270700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:00.270788 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:00.271460 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:53:00.271474 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:53:00.273040 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:53:00.275660 (+ 2620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:00.275665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:00.275668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:00.276100 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:00.277051 (+ 951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:00.277055 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:00.277057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:00.277168 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:00.277631 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:53:00.277634 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:53:00.277738 (+ 104us) client_negotiation.cc:769] Sending connection context
0811 11:53:00.277833 (+ 95us) client_negotiation.cc:240] Negotiation successful
0811 11:53:00.278042 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260811 11:53:00.282299 25730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948: Bootstrap starting.
I20260811 11:53:00.286633 25730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948: Neither blocks nor log segments found. Creating new log.
I20260811 11:53:00.287729 25730 log.cc:824] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948: Log is configured to *not* fsync() on all Append() calls
I20260811 11:53:00.291011 25730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948: No bootstrap required, opened a new log
I20260811 11:53:00.294549 25730 raft_consensus.cc:348] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1124c123494619a6294d6f0db4e948" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38717 } } peers { permanent_uuid: "d7952c86f2cc4827ac9e3b6940ec9054" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } } peers { permanent_uuid: "395fd6ac31db4c4aa65b945125d12e2d" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } }
I20260811 11:53:00.294812 25730 raft_consensus.cc:374] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:53:00.294864 25730 raft_consensus.cc:729] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1124c123494619a6294d6f0db4e948, State: Initialized, Role: FOLLOWER
I20260811 11:53:00.295359 25730 consensus_queue.cc:260] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1124c123494619a6294d6f0db4e948" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38717 } } peers { permanent_uuid: "d7952c86f2cc4827ac9e3b6940ec9054" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } } peers { permanent_uuid: "395fd6ac31db4c4aa65b945125d12e2d" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } }
I20260811 11:53:00.295943 26027 sys_catalog.cc:455] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1124c123494619a6294d6f0db4e948" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38717 } } peers { permanent_uuid: "d7952c86f2cc4827ac9e3b6940ec9054" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } } peers { permanent_uuid: "395fd6ac31db4c4aa65b945125d12e2d" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } } }
I20260811 11:53:00.296267 26027 sys_catalog.cc:458] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:53:00.297127 25730 sys_catalog.cc:564] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:53:00.299600 25851 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } attempt: 4
W20260811 11:53:00.304869 26046 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:53:00.305053 26046 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:53:00.302922 25620 master_runner.cc:186] Error getting master registration for 127.3.221.125:39703: OK, instance_id { permanent_uuid: "d7952c86f2cc4827ac9e3b6940ec9054" instance_seqno: 1786449179890307 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 11:53:00.307220 25620 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 11 11:53:00 dist-test-slave-bndk krb5kdc[25600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449179, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.125@KRBTEST.COM for kudu/127.3.221.124@KRBTEST.COM
I20260811 11:53:00.312311 25854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.300110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:33769 (local address 127.0.0.1:43620)
0811 11:53:00.300412 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:00.300428 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:00.300523 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:00.301967 (+ 1444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:00.301970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:00.302451 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:53:00.302589 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:00.302599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.303843 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.303847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:00.304647 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:00.304662 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.305448 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.305451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:00.305553 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:00.306133 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:53:00.306149 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:53:00.307698 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:53:00.309872 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:00.309877 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:00.309880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:00.310299 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:00.310707 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:00.310710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:00.310712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:00.310868 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:00.311527 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:53:00.311530 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:53:00.311728 (+ 198us) client_negotiation.cc:769] Sending connection context
0811 11:53:00.311855 (+ 127us) client_negotiation.cc:240] Negotiation successful
0811 11:53:00.312085 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260811 11:53:00.313060 25980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:00.300937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43620 (local address 127.3.221.124:33769)
0811 11:53:00.301306 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:00.301311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:00.301326 (+ 15us) server_negotiation.cc:407] Connection header received
0811 11:53:00.301567 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:00.301572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:00.301667 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:00.301778 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:00.302911 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.303691 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.304805 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:00.305140 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:00.305299 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:00.307869 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:00.307872 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:00.307874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:00.307900 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:00.309637 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.310467 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.310471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.310473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.310588 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:00.311018 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:00.311022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:00.311024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:00.311354 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:00.311456 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:00.312659 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0811 11:53:00.312838 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269}
I20260811 11:53:00.316569 25851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054: Bootstrap starting.
I20260811 11:53:00.319300 25851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054: Neither blocks nor log segments found. Creating new log.
I20260811 11:53:00.320113 25851 log.cc:824] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054: Log is configured to *not* fsync() on all Append() calls
I20260811 11:53:00.322364 25851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054: No bootstrap required, opened a new log
I20260811 11:53:00.325707 25851 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1124c123494619a6294d6f0db4e948" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38717 } } peers { permanent_uuid: "d7952c86f2cc4827ac9e3b6940ec9054" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } } peers { permanent_uuid: "395fd6ac31db4c4aa65b945125d12e2d" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } }
I20260811 11:53:00.325955 25851 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:53:00.326010 25851 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7952c86f2cc4827ac9e3b6940ec9054, State: Initialized, Role: FOLLOWER
I20260811 11:53:00.326485 25851 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1124c123494619a6294d6f0db4e948" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38717 } } peers { permanent_uuid: "d7952c86f2cc4827ac9e3b6940ec9054" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } } peers { permanent_uuid: "395fd6ac31db4c4aa65b945125d12e2d" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } }
I20260811 11:53:00.327113 26053 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1124c123494619a6294d6f0db4e948" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38717 } } peers { permanent_uuid: "d7952c86f2cc4827ac9e3b6940ec9054" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } } peers { permanent_uuid: "395fd6ac31db4c4aa65b945125d12e2d" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } } }
I20260811 11:53:00.327324 26053 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:53:00.328186 25851 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054 [sys.catalog]: configured and running, proceeding with master startup.
W20260811 11:53:00.334581 26071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:53:00.334791 26071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:53:00.338291 25739 master_runner.cc:263] No leader master found from master d7952c86f2cc4827ac9e3b6940ec9054
I20260811 11:53:00.338394 25739 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260811 11:53:00.378841 25657 tablet.cc:2369] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:53:00.386915 25773 tablet.cc:2369] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:53:00.397773 26027 raft_consensus.cc:482] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:53:00.398078 26027 raft_consensus.cc:504] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1124c123494619a6294d6f0db4e948" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38717 } } peers { permanent_uuid: "d7952c86f2cc4827ac9e3b6940ec9054" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } } peers { permanent_uuid: "395fd6ac31db4c4aa65b945125d12e2d" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } }
I20260811 11:53:00.399343 26027 leader_election.cc:288] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7952c86f2cc4827ac9e3b6940ec9054 (127.3.221.125:39703), 395fd6ac31db4c4aa65b945125d12e2d (127.3.221.124:33769)
I20260811 11:53:00.400323 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "be1124c123494619a6294d6f0db4e948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395fd6ac31db4c4aa65b945125d12e2d" is_pre_election: true
I20260811 11:53:00.400734 25931 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be1124c123494619a6294d6f0db4e948 in term 0.
I20260811 11:53:00.402055 25652 leader_election.cc:302] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 395fd6ac31db4c4aa65b945125d12e2d, be1124c123494619a6294d6f0db4e948; no voters:
I20260811 11:53:00.402937 25816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "be1124c123494619a6294d6f0db4e948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7952c86f2cc4827ac9e3b6940ec9054" is_pre_election: true
I20260811 11:53:00.403313 25816 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be1124c123494619a6294d6f0db4e948 in term 0.
I20260811 11:53:00.404150 26027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:53:00.404273 26027 raft_consensus.cc:482] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:53:00.404331 26027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:53:00.406276 26027 raft_consensus.cc:504] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1124c123494619a6294d6f0db4e948" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38717 } } peers { permanent_uuid: "d7952c86f2cc4827ac9e3b6940ec9054" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } } peers { permanent_uuid: "395fd6ac31db4c4aa65b945125d12e2d" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } }
I20260811 11:53:00.407150 25816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "be1124c123494619a6294d6f0db4e948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7952c86f2cc4827ac9e3b6940ec9054"
I20260811 11:53:00.407310 25816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:53:00.408730 25816 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be1124c123494619a6294d6f0db4e948 in term 1.
I20260811 11:53:00.409314 25652 leader_election.cc:302] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be1124c123494619a6294d6f0db4e948, d7952c86f2cc4827ac9e3b6940ec9054; no voters:
I20260811 11:53:00.412614 26027 leader_election.cc:288] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [CANDIDATE]: Term 1 election: Requested vote from peers d7952c86f2cc4827ac9e3b6940ec9054 (127.3.221.125:39703), 395fd6ac31db4c4aa65b945125d12e2d (127.3.221.124:33769)
I20260811 11:53:00.413182 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "be1124c123494619a6294d6f0db4e948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395fd6ac31db4c4aa65b945125d12e2d"
I20260811 11:53:00.413380 25931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d [term 0 FOLLOWER]: Advancing to term 1
W20260811 11:53:00.414458 25899 tablet.cc:2369] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:53:00.414951 25931 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be1124c123494619a6294d6f0db4e948 in term 1.
I20260811 11:53:00.415479 26027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:53:00.415611 26027 raft_consensus.cc:686] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [term 1 LEADER]: Becoming Leader. State: Replica: be1124c123494619a6294d6f0db4e948, State: Running, Role: LEADER
I20260811 11:53:00.416015 26027 consensus_queue.cc:237] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1124c123494619a6294d6f0db4e948" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38717 } } peers { permanent_uuid: "d7952c86f2cc4827ac9e3b6940ec9054" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } } peers { permanent_uuid: "395fd6ac31db4c4aa65b945125d12e2d" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } }
I20260811 11:53:00.418035 26027 sys_catalog.cc:455] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be1124c123494619a6294d6f0db4e948. Latest consensus state: current_term: 1 leader_uuid: "be1124c123494619a6294d6f0db4e948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1124c123494619a6294d6f0db4e948" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38717 } } peers { permanent_uuid: "d7952c86f2cc4827ac9e3b6940ec9054" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } } peers { permanent_uuid: "395fd6ac31db4c4aa65b945125d12e2d" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } } }
I20260811 11:53:00.418175 26027 sys_catalog.cc:458] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [sys.catalog]: This master's current role is: LEADER
I20260811 11:53:00.420181 26076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:53:00.421330 26076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:53:00.427981 25931 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d [term 1 FOLLOWER]: Refusing update from remote peer be1124c123494619a6294d6f0db4e948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:53:00.431407 26074 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395fd6ac31db4c4aa65b945125d12e2d" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:53:00.436493 26000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader be1124c123494619a6294d6f0db4e948. Latest consensus state: current_term: 1 leader_uuid: "be1124c123494619a6294d6f0db4e948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1124c123494619a6294d6f0db4e948" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38717 } } peers { permanent_uuid: "d7952c86f2cc4827ac9e3b6940ec9054" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } } peers { permanent_uuid: "395fd6ac31db4c4aa65b945125d12e2d" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } } }
I20260811 11:53:00.436655 26000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:53:00.437892 25816 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054 [term 1 FOLLOWER]: Refusing update from remote peer be1124c123494619a6294d6f0db4e948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:53:00.438542 26074 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7952c86f2cc4827ac9e3b6940ec9054" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:53:00.443147 26053 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be1124c123494619a6294d6f0db4e948. Latest consensus state: current_term: 1 leader_uuid: "be1124c123494619a6294d6f0db4e948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1124c123494619a6294d6f0db4e948" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38717 } } peers { permanent_uuid: "d7952c86f2cc4827ac9e3b6940ec9054" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } } peers { permanent_uuid: "395fd6ac31db4c4aa65b945125d12e2d" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } } }
I20260811 11:53:00.443324 26053 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:53:00.447127 26074 sys_catalog.cc:455] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be1124c123494619a6294d6f0db4e948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1124c123494619a6294d6f0db4e948" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38717 } } peers { permanent_uuid: "d7952c86f2cc4827ac9e3b6940ec9054" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } } peers { permanent_uuid: "395fd6ac31db4c4aa65b945125d12e2d" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } } }
I20260811 11:53:00.447283 26074 sys_catalog.cc:458] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [sys.catalog]: This master's current role is: LEADER
I20260811 11:53:00.448500 26053 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be1124c123494619a6294d6f0db4e948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1124c123494619a6294d6f0db4e948" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38717 } } peers { permanent_uuid: "d7952c86f2cc4827ac9e3b6940ec9054" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } } peers { permanent_uuid: "395fd6ac31db4c4aa65b945125d12e2d" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } } }
I20260811 11:53:00.448653 26053 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:53:00.449265 26077 mvcc.cc:205] Tried to move back new op lower bound from 7317295843023659008 to 7317295842987483136. Current Snapshot: MvccSnapshot[applied={T|T < 7317295843023659008}]
I20260811 11:53:00.449891 26027 sys_catalog.cc:455] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be1124c123494619a6294d6f0db4e948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1124c123494619a6294d6f0db4e948" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38717 } } peers { permanent_uuid: "d7952c86f2cc4827ac9e3b6940ec9054" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } } peers { permanent_uuid: "395fd6ac31db4c4aa65b945125d12e2d" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } } }
I20260811 11:53:00.450034 26027 sys_catalog.cc:458] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948 [sys.catalog]: This master's current role is: LEADER
I20260811 11:53:00.450942 26080 mvcc.cc:205] Tried to move back new op lower bound from 7317295843023659008 to 7317295842987483136. Current Snapshot: MvccSnapshot[applied={T|T < 7317295843023659008}]
I20260811 11:53:00.452729 26076 catalog_manager.cc:1347] Generated new cluster ID: 31be5c7596de4ba6b27532859c444d7e
I20260811 11:53:00.452893 26076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:53:00.457056 26000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be1124c123494619a6294d6f0db4e948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1124c123494619a6294d6f0db4e948" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 38717 } } peers { permanent_uuid: "d7952c86f2cc4827ac9e3b6940ec9054" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 39703 } } peers { permanent_uuid: "395fd6ac31db4c4aa65b945125d12e2d" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 33769 } } }
I20260811 11:53:00.457260 26000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:53:00.473868 26076 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:53:00.475317 26076 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:53:00.490587 26076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be1124c123494619a6294d6f0db4e948: Generated new TSK 0
I20260811 11:53:00.491699 26076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:53:01.244524 26022 catalog_manager.cc:1259] Loaded cluster ID: 31be5c7596de4ba6b27532859c444d7e
I20260811 11:53:01.244675 26022 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d: loading cluster ID for follower catalog manager: success
I20260811 11:53:01.246937 26022 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d: acquiring CA information for follower catalog manager: success
I20260811 11:53:01.248349 26022 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 395fd6ac31db4c4aa65b945125d12e2d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 11:53:01.336390 26071 catalog_manager.cc:1259] Loaded cluster ID: 31be5c7596de4ba6b27532859c444d7e
I20260811 11:53:01.336491 26071 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054: loading cluster ID for follower catalog manager: success
I20260811 11:53:01.338521 26071 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054: acquiring CA information for follower catalog manager: success
I20260811 11:53:01.339825 26071 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d7952c86f2cc4827ac9e3b6940ec9054: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 11:53:01.564615 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 25620
I20260811 11:53:01.577632 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 25739
I20260811 11:53:01.590204 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 25858
2026-08-11T11:53:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3493 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:53:01 dist-test-slave-bndk krb5kdc[26099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:53:01 dist-test-slave-bndk krb5kdc[26099](info): set up 2 sockets
Aug 11 11:53:01 dist-test-slave-bndk krb5kdc[26099](info): commencing operation
krb5kdc: starting...
W20260811 11:53:02.799129 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:53:02 dist-test-slave-bndk krb5kdc[26099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:53:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:53:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:53:02.945340 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:33421
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:33421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:53:03.068678 26123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:53:03.069005 26123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:53:03.069049 26123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:53:03.073570 26123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:53:03.073652 26123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:53:03.073673 26123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:53:03.073691 26123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:53:03.073706 26123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:53:03.079627 26123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:33421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:33421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:53:03.080869 26123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:53:03.082146 26123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:53:03.091284 26136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:03.091300 26130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:03.091310 26132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:03.091384 26123 server_base.cc:1034] running on GCE node
I20260811 11:53:03.092182 26123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:53:03.093464 26123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:53:03.094661 26123 hybrid_clock.cc:648] HybridClock initialized: now 1786449183094633 us; error 41 us; skew 500 ppm
Aug 11 11:53:03 dist-test-slave-bndk krb5kdc[26099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449183, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:53:03.097807 26123 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:53:03.098847 26123 webserver.cc:460] Webserver started at http://127.3.221.126:40857/ using document root <none> and password file <none>
I20260811 11:53:03.099458 26123 fs_manager.cc:359] Metadata directory not provided
I20260811 11:53:03.099529 26123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:53:03.099747 26123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:53:03.102072 26123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "177c332fcc5740b9a16172292090b444"
format_stamp: "Formatted at 2026-08-11 11:53:03 on dist-test-slave-bndk"
server_key: "9ef3a656f3f7b4530e4dc34b8b4956ec"
server_key_iv: "79afbcd9d93e46ccd25ecac008e4bec3"
server_key_version: "encryptionkey@0"
I20260811 11:53:03.102677 26123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "177c332fcc5740b9a16172292090b444"
format_stamp: "Formatted at 2026-08-11 11:53:03 on dist-test-slave-bndk"
server_key: "9ef3a656f3f7b4530e4dc34b8b4956ec"
server_key_iv: "79afbcd9d93e46ccd25ecac008e4bec3"
server_key_version: "encryptionkey@0"
I20260811 11:53:03.107194 26123 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 11:53:03.110502 26146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:03.112437 26123 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260811 11:53:03.112600 26123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "177c332fcc5740b9a16172292090b444"
format_stamp: "Formatted at 2026-08-11 11:53:03 on dist-test-slave-bndk"
server_key: "9ef3a656f3f7b4530e4dc34b8b4956ec"
server_key_iv: "79afbcd9d93e46ccd25ecac008e4bec3"
server_key_version: "encryptionkey@0"
I20260811 11:53:03.112759 26123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:53:03.131250 26123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:53:03.134733 26123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:53:03.134938 26123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:53:03.142501 26123 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:33421
I20260811 11:53:03.142558 26231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:33421 every 8 connection(s)
I20260811 11:53:03.143543 26123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:53:03.144008 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 26123
I20260811 11:53:03.144152 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:53:03.144460 3957 external_mini_cluster.cc:1428] Setting key b4d98c7cd9dd9e792467e961a1637cc6
I20260811 11:53:03.147446 26232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 11 11:53:03 dist-test-slave-bndk krb5kdc[26099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:53:03.154587 26232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444: Bootstrap starting.
I20260811 11:53:03.157444 26232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444: Neither blocks nor log segments found. Creating new log.
I20260811 11:53:03.158394 26232 log.cc:824] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444: Log is configured to *not* fsync() on all Append() calls
I20260811 11:53:03.160436 26240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:03.146317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37186 (local address 127.3.221.126:33421)
0811 11:53:03.146840 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:03.146857 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:03.146901 (+ 44us) server_negotiation.cc:407] Connection header received
0811 11:53:03.147655 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:03.147675 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:03.148041 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:03.148362 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:03.149372 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:03.150435 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:03.151639 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:03.152155 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:03.152288 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:03.155372 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:03.155378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:03.155393 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:03.155429 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:03.157278 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:03.157820 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:03.157826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:03.157832 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:03.157969 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:03.158261 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:03.158263 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:03.158264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:03.158718 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:03.158905 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:03.159204 (+ 299us) server_negotiation.cc:299] Negotiation successful
0811 11:53:03.159497 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":156,"threads_started":1}
I20260811 11:53:03.160765 26232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444: No bootstrap required, opened a new log
I20260811 11:53:03.163847 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:53:03.165138 26232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177c332fcc5740b9a16172292090b444" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33421 } }
I20260811 11:53:03.165484 26232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:53:03.165539 26232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 177c332fcc5740b9a16172292090b444, State: Initialized, Role: FOLLOWER
I20260811 11:53:03.165979 26232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177c332fcc5740b9a16172292090b444" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33421 } }
I20260811 11:53:03.166118 26232 raft_consensus.cc:388] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:53:03.166194 26232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:53:03.166301 26232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:53:03.167338 26232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177c332fcc5740b9a16172292090b444" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33421 } }
I20260811 11:53:03.167706 26232 leader_election.cc:302] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 177c332fcc5740b9a16172292090b444; no voters:
I20260811 11:53:03.168044 26232 leader_election.cc:288] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:53:03.168239 26242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:53:03.168505 26242 raft_consensus.cc:686] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444 [term 1 LEADER]: Becoming Leader. State: Replica: 177c332fcc5740b9a16172292090b444, State: Running, Role: LEADER
I20260811 11:53:03.168821 26242 consensus_queue.cc:237] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177c332fcc5740b9a16172292090b444" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33421 } }
I20260811 11:53:03.169446 26232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:53:03.170522 26247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 177c332fcc5740b9a16172292090b444. Latest consensus state: current_term: 1 leader_uuid: "177c332fcc5740b9a16172292090b444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177c332fcc5740b9a16172292090b444" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33421 } } }
I20260811 11:53:03.170665 26247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444 [sys.catalog]: This master's current role is: LEADER
I20260811 11:53:03.170948 26243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "177c332fcc5740b9a16172292090b444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177c332fcc5740b9a16172292090b444" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33421 } } }
I20260811 11:53:03.171048 26243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444 [sys.catalog]: This master's current role is: LEADER
W20260811 11:53:03.174633 26264 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:53:03.174717 26264 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:53:03.174804 26256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:53:03.175544 26256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:53:03.180904 26256 catalog_manager.cc:1347] Generated new cluster ID: 78a64536d3c14e8697a3110637649c8f
I20260811 11:53:03.180994 26256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:53:03.216926 26256 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:53:03.218106 26256 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:53:03.223037 26256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 177c332fcc5740b9a16172292090b444: Generated new TSK 0
I20260811 11:53:03.223815 26256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:53:03.272151 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 26123
2026-08-11T11:53:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1678 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:53:03 dist-test-slave-bndk krb5kdc[26279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:53:03 dist-test-slave-bndk krb5kdc[26279](info): set up 2 sockets
Aug 11 11:53:03 dist-test-slave-bndk krb5kdc[26279](info): commencing operation
krb5kdc: starting...
W20260811 11:53:04.503405 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:53:04 dist-test-slave-bndk krb5kdc[26279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:53:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:53:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:53:04.656976 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:40419
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:40419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:53:04.785586 26304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:53:04.786043 26304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:53:04.786110 26304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:53:04.790800 26304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:53:04.790900 26304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:53:04.790934 26304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:53:04.790957 26304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:53:04.790979 26304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:53:04.797055 26304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:40419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:40419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:53:04.798385 26304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:53:04.799540 26304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:53:04.809099 26312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:04.809345 26313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:04.809358 26317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:04.809691 26304 server_base.cc:1034] running on GCE node
I20260811 11:53:04.811022 26304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:53:04.812444 26304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:53:04.815786 26304 hybrid_clock.cc:648] HybridClock initialized: now 1786449184815761 us; error 54 us; skew 500 ppm
Aug 11 11:53:04 dist-test-slave-bndk krb5kdc[26279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449184, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:53:04.819017 26304 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:53:04.820155 26304 webserver.cc:460] Webserver started at http://127.3.221.126:43831/ using document root <none> and password file <none>
I20260811 11:53:04.820748 26304 fs_manager.cc:359] Metadata directory not provided
I20260811 11:53:04.820818 26304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:53:04.821028 26304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:53:04.823318 26304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "691ca47614cf4870a72b077bdc797e82"
format_stamp: "Formatted at 2026-08-11 11:53:04 on dist-test-slave-bndk"
server_key: "830487391d2e4b55a211fd099941d060"
server_key_iv: "c8ef99263eaba4e050d7ad6383ad42ee"
server_key_version: "encryptionkey@0"
I20260811 11:53:04.823925 26304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "691ca47614cf4870a72b077bdc797e82"
format_stamp: "Formatted at 2026-08-11 11:53:04 on dist-test-slave-bndk"
server_key: "830487391d2e4b55a211fd099941d060"
server_key_iv: "c8ef99263eaba4e050d7ad6383ad42ee"
server_key_version: "encryptionkey@0"
I20260811 11:53:04.828100 26304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260811 11:53:04.831007 26326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:04.832499 26304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 11:53:04.832636 26304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "691ca47614cf4870a72b077bdc797e82"
format_stamp: "Formatted at 2026-08-11 11:53:04 on dist-test-slave-bndk"
server_key: "830487391d2e4b55a211fd099941d060"
server_key_iv: "c8ef99263eaba4e050d7ad6383ad42ee"
server_key_version: "encryptionkey@0"
I20260811 11:53:04.832751 26304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:53:04.858060 26304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:53:04.861493 26304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:53:04.861693 26304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:53:04.869341 26304 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:40419
I20260811 11:53:04.869633 26419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:40419 every 8 connection(s)
I20260811 11:53:04.870456 26304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:53:04.873613 26420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:53:04.875730 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 26304
I20260811 11:53:04.875849 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:53:04.876174 3957 external_mini_cluster.cc:1428] Setting key a92ead133704617f883bd723b36bfa4a
I20260811 11:53:04.880640 26420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82: Bootstrap starting.
I20260811 11:53:04.883559 26420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82: Neither blocks nor log segments found. Creating new log.
I20260811 11:53:04.884658 26420 log.cc:824] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82: Log is configured to *not* fsync() on all Append() calls
Aug 11 11:53:04 dist-test-slave-bndk krb5kdc[26279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:53:04.887058 26420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82: No bootstrap required, opened a new log
I20260811 11:53:04.890488 26420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691ca47614cf4870a72b077bdc797e82" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } }
I20260811 11:53:04.890762 26420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:53:04.890799 26420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691ca47614cf4870a72b077bdc797e82, State: Initialized, Role: FOLLOWER
I20260811 11:53:04.891286 26420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691ca47614cf4870a72b077bdc797e82" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } }
I20260811 11:53:04.891433 26420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:53:04.891510 26420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:53:04.891619 26420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:53:04.892840 26420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691ca47614cf4870a72b077bdc797e82" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } }
I20260811 11:53:04.893260 26420 leader_election.cc:302] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 691ca47614cf4870a72b077bdc797e82; no voters:
I20260811 11:53:04.893697 26420 leader_election.cc:288] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:53:04.893851 26431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:53:04.894290 26431 raft_consensus.cc:686] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 1 LEADER]: Becoming Leader. State: Replica: 691ca47614cf4870a72b077bdc797e82, State: Running, Role: LEADER
I20260811 11:53:04.894672 26431 consensus_queue.cc:237] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691ca47614cf4870a72b077bdc797e82" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } }
I20260811 11:53:04.895118 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:04.877794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58786 (local address 127.3.221.126:40419)
0811 11:53:04.878267 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:04.878288 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:04.878334 (+ 46us) server_negotiation.cc:407] Connection header received
0811 11:53:04.879362 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:04.879388 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:04.879853 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:04.880256 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:04.880992 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:04.882581 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:04.883525 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:04.884308 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:04.884506 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:04.887691 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:04.887697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:04.887713 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:04.887744 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:04.891332 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:04.891973 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:04.891979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:04.891984 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:04.892123 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:04.892473 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:04.892476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:04.892478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:04.892960 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:04.893179 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:04.893679 (+ 500us) server_negotiation.cc:299] Negotiation successful
0811 11:53:04.894063 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":126,"threads_started":1}
I20260811 11:53:04.895226 26420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:53:04.896421 26432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "691ca47614cf4870a72b077bdc797e82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691ca47614cf4870a72b077bdc797e82" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } } }
I20260811 11:53:04.896559 26432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [sys.catalog]: This master's current role is: LEADER
I20260811 11:53:04.897452 26433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 691ca47614cf4870a72b077bdc797e82. Latest consensus state: current_term: 1 leader_uuid: "691ca47614cf4870a72b077bdc797e82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691ca47614cf4870a72b077bdc797e82" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } } }
I20260811 11:53:04.897617 26433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [sys.catalog]: This master's current role is: LEADER
W20260811 11:53:04.900467 26451 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:53:04.900549 26451 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:53:04.900952 26444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:53:04.901832 26444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:53:04.907989 26444 catalog_manager.cc:1347] Generated new cluster ID: 94cac514bcb94264a4c93de5648b35ba
I20260811 11:53:04.908082 26444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:53:04.928194 26444 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:53:04.929284 26444 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:53:04.936640 26444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82: Generated new TSK 0
I20260811 11:53:04.937546 26444 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
I20260811 11:53:04.981304 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:40419
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:53:05.110306 26461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:53:05.110603 26461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:53:05.110649 26461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:53:05.115140 26461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:53:05.115254 26461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:53:05.115327 26461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:53:05.121070 26461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:40419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:53:05.122316 26461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:53:05.123432 26461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:53:05.134020 26469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:05.133991 26471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:05.133991 26480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:05.134491 26461 server_base.cc:1034] running on GCE node
I20260811 11:53:05.134979 26461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:53:05.135715 26461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:53:05.136929 26461 hybrid_clock.cc:648] HybridClock initialized: now 1786449185136904 us; error 34 us; skew 500 ppm
Aug 11 11:53:05 dist-test-slave-bndk krb5kdc[26279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449185, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:53:05.140154 26461 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:53:05.141178 26461 webserver.cc:460] Webserver started at http://127.3.221.65:42685/ using document root <none> and password file <none>
I20260811 11:53:05.141845 26461 fs_manager.cc:359] Metadata directory not provided
I20260811 11:53:05.141932 26461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:53:05.142159 26461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:53:05.144371 26461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "bc6b1d1884454450ae1f9cbc66519a24"
format_stamp: "Formatted at 2026-08-11 11:53:05 on dist-test-slave-bndk"
server_key: "c92c5d7a16739b8ed3323f88351d5264"
server_key_iv: "310a2536254248c76216b2c8eb1e99ed"
server_key_version: "encryptionkey@0"
I20260811 11:53:05.144982 26461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "bc6b1d1884454450ae1f9cbc66519a24"
format_stamp: "Formatted at 2026-08-11 11:53:05 on dist-test-slave-bndk"
server_key: "c92c5d7a16739b8ed3323f88351d5264"
server_key_iv: "310a2536254248c76216b2c8eb1e99ed"
server_key_version: "encryptionkey@0"
I20260811 11:53:05.149489 26461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:53:05.153080 26491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:05.154922 26461 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 11:53:05.155082 26461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "bc6b1d1884454450ae1f9cbc66519a24"
format_stamp: "Formatted at 2026-08-11 11:53:05 on dist-test-slave-bndk"
server_key: "c92c5d7a16739b8ed3323f88351d5264"
server_key_iv: "310a2536254248c76216b2c8eb1e99ed"
server_key_version: "encryptionkey@0"
I20260811 11:53:05.155197 26461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:53:05.193564 26461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:53:05.197445 26461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:53:05.197647 26461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:53:05.198333 26461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:53:05.199267 26461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:53:05.199316 26461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:05.199358 26461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:53:05.199388 26461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:05.209683 26461 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:32839
I20260811 11:53:05.209761 26678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:32839 every 8 connection(s)
I20260811 11:53:05.210728 26461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:53:05.211648 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 26461
I20260811 11:53:05.211812 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:53:05.212121 3957 external_mini_cluster.cc:1428] Setting key e30677503c59b1a4f91815a21f37784e
Aug 11 11:53:05 dist-test-slave-bndk krb5kdc[26279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449185, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.66@KRBTEST.COM" created.
I20260811 11:53:05.228744 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:05.212931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:49523 (local address 127.3.221.126:40419)
0811 11:53:05.213057 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:05.213061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:05.214411 (+ 1350us) server_negotiation.cc:407] Connection header received
0811 11:53:05.215341 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:05.215344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:05.215402 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:05.215491 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:05.216898 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.217999 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.219489 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.219748 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.219847 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:05.222955 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:05.222959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:05.222961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:05.222995 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:05.225031 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:05.226135 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:05.226139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:05.226140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:05.226273 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:05.226739 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:05.226742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:05.226743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:05.227175 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:05.227362 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:05.228320 (+ 958us) server_negotiation.cc:299] Negotiation successful
0811 11:53:05.228538 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 11:53:05.230255 26681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:05.213427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.65:49523)
0811 11:53:05.214215 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:05.214283 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:05.215125 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:05.215662 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:05.215670 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:05.216061 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:53:05.216597 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:05.216612 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.218197 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.218200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:05.219304 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:05.219318 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.219927 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.219931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:05.220123 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:05.220816 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:53:05.220839 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:53:05.222730 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:53:05.225280 (+ 2550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:05.225292 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:05.225324 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:05.225960 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:05.226389 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:05.226392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:05.226395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:05.226622 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:05.227457 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:53:05.227468 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:53:05.227907 (+ 439us) client_negotiation.cc:769] Sending connection context
0811 11:53:05.228147 (+ 240us) client_negotiation.cc:240] Negotiation successful
0811 11:53:05.228585 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":140,"threads_started":1}
I20260811 11:53:05.231948 26679 heartbeater.cc:347] Connected to a master server at 127.3.221.126:40419
I20260811 11:53:05.232246 26679 heartbeater.cc:464] Registering TS with master...
I20260811 11:53:05.232909 26679 heartbeater.cc:511] Master 127.3.221.126:40419 requested a full tablet report, sending...
I20260811 11:53:05.235008 26359 ts_manager.cc:194] Registered new tserver with Master: bc6b1d1884454450ae1f9cbc66519a24 (127.3.221.65:32839)
I20260811 11:53:05.236720 26359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:49523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
I20260811 11:53:05.250365 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:40419
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:53:05.376525 26687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:53:05.376858 26687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:53:05.376905 26687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:53:05.381279 26687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:53:05.381368 26687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:53:05.381431 26687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:53:05.387140 26687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:40419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:53:05.388387 26687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:53:05.389518 26687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:53:05.400329 26698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:05.400350 26697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:05.400309 26705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:05.400943 26687 server_base.cc:1034] running on GCE node
I20260811 11:53:05.401693 26687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:53:05.402431 26687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:53:05.403631 26687 hybrid_clock.cc:648] HybridClock initialized: now 1786449185403574 us; error 74 us; skew 500 ppm
Aug 11 11:53:05 dist-test-slave-bndk krb5kdc[26279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449185, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:53:05.407436 26687 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:53:05.408463 26687 webserver.cc:460] Webserver started at http://127.3.221.66:34647/ using document root <none> and password file <none>
I20260811 11:53:05.409089 26687 fs_manager.cc:359] Metadata directory not provided
I20260811 11:53:05.409157 26687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:53:05.409432 26687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:53:05.411726 26687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "903b03d331eb406da01ced709713f1fe"
format_stamp: "Formatted at 2026-08-11 11:53:05 on dist-test-slave-bndk"
server_key: "65bd0c4a5c31add6f734b499f31f9aba"
server_key_iv: "ddcab6a90e1c315d970cc78d183a9f6a"
server_key_version: "encryptionkey@0"
I20260811 11:53:05.412367 26687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "903b03d331eb406da01ced709713f1fe"
format_stamp: "Formatted at 2026-08-11 11:53:05 on dist-test-slave-bndk"
server_key: "65bd0c4a5c31add6f734b499f31f9aba"
server_key_iv: "ddcab6a90e1c315d970cc78d183a9f6a"
server_key_version: "encryptionkey@0"
I20260811 11:53:05.417582 26687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 11:53:05.420918 26721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:05.422812 26687 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 11:53:05.422971 26687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "903b03d331eb406da01ced709713f1fe"
format_stamp: "Formatted at 2026-08-11 11:53:05 on dist-test-slave-bndk"
server_key: "65bd0c4a5c31add6f734b499f31f9aba"
server_key_iv: "ddcab6a90e1c315d970cc78d183a9f6a"
server_key_version: "encryptionkey@0"
I20260811 11:53:05.423097 26687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:53:05.454084 26687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:53:05.457605 26687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:53:05.457808 26687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:53:05.458475 26687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:53:05.459477 26687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:53:05.459547 26687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:05.459595 26687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:53:05.459611 26687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:05.469717 26687 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:41217
I20260811 11:53:05.469812 26890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:41217 every 8 connection(s)
I20260811 11:53:05.470693 26687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:53:05.480141 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 26687
I20260811 11:53:05.480285 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:53:05.480602 3957 external_mini_cluster.cc:1428] Setting key 4f972660761b87fcdd1e9eb3d935b090
Aug 11 11:53:05 dist-test-slave-bndk krb5kdc[26279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449185, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:53:05.487155 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:05.473051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:46131 (local address 127.3.221.126:40419)
0811 11:53:05.473184 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:05.473188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:05.474117 (+ 929us) server_negotiation.cc:407] Connection header received
0811 11:53:05.475531 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:05.475535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:05.475642 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:05.475772 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:05.477290 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.477995 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.479353 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.479635 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.479768 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:05.483438 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:05.483442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:05.483444 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:05.483470 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:05.484873 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:05.485495 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:05.485500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:05.485501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:05.485577 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:05.485924 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:05.485927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:05.485928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:05.486166 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:05.486257 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:05.486830 (+ 573us) server_negotiation.cc:299] Negotiation successful
0811 11:53:05.486981 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 11:53:05.488055 26898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:05.473416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.66:46131)
0811 11:53:05.473952 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:05.474000 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:05.475206 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:05.476010 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:05.476018 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:05.476436 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:53:05.476996 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:05.477012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.478196 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.478200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:05.479170 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:05.479182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.479877 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.479884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:05.480110 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:05.480836 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:53:05.480872 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:53:05.483180 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:53:05.485012 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:05.485021 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:05.485041 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:05.485396 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:05.485665 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:05.485668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:05.485670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:05.485824 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:05.486273 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:53:05.486281 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:53:05.486567 (+ 286us) client_negotiation.cc:769] Sending connection context
0811 11:53:05.486746 (+ 179us) client_negotiation.cc:240] Negotiation successful
0811 11:53:05.487116 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":117,"threads_started":1}
I20260811 11:53:05.489285 26894 heartbeater.cc:347] Connected to a master server at 127.3.221.126:40419
I20260811 11:53:05.489585 26894 heartbeater.cc:464] Registering TS with master...
I20260811 11:53:05.490250 26894 heartbeater.cc:511] Master 127.3.221.126:40419 requested a full tablet report, sending...
I20260811 11:53:05.491628 26359 ts_manager.cc:194] Registered new tserver with Master: 903b03d331eb406da01ced709713f1fe (127.3.221.66:41217)
I20260811 11:53:05.492264 26359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:46131
WARNING: no policy specified for kudu/127.3.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
I20260811 11:53:05.517553 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:40419
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:53:05.650002 26902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:53:05.650331 26902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:53:05.650396 26902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:53:05.654822 26902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:53:05.654906 26902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:53:05.654991 26902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:53:05.661043 26902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:40419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:53:05.662390 26902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:53:05.663473 26902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:53:05.673607 26909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:05.673691 26911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:05.673617 26902 server_base.cc:1034] running on GCE node
W20260811 11:53:05.673604 26908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:05.674358 26902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:53:05.675042 26902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:53:05.676236 26902 hybrid_clock.cc:648] HybridClock initialized: now 1786449185676197 us; error 51 us; skew 500 ppm
Aug 11 11:53:05 dist-test-slave-bndk krb5kdc[26279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449185, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:53:05.679562 26902 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:53:05.680584 26902 webserver.cc:460] Webserver started at http://127.3.221.67:45853/ using document root <none> and password file <none>
I20260811 11:53:05.681190 26902 fs_manager.cc:359] Metadata directory not provided
I20260811 11:53:05.681298 26902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:53:05.681532 26902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:53:05.683880 26902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "0efb25eba2184dbfb0e476a285fef172"
format_stamp: "Formatted at 2026-08-11 11:53:05 on dist-test-slave-bndk"
server_key: "05657cc35ad863404415f85d762e2cc1"
server_key_iv: "03e9aaac1fb55650fcd1afbd8330ee25"
server_key_version: "encryptionkey@0"
I20260811 11:53:05.684490 26902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "0efb25eba2184dbfb0e476a285fef172"
format_stamp: "Formatted at 2026-08-11 11:53:05 on dist-test-slave-bndk"
server_key: "05657cc35ad863404415f85d762e2cc1"
server_key_iv: "03e9aaac1fb55650fcd1afbd8330ee25"
server_key_version: "encryptionkey@0"
I20260811 11:53:05.688655 26902 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 11:53:05.693087 26923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:05.694859 26902 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260811 11:53:05.695003 26902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "0efb25eba2184dbfb0e476a285fef172"
format_stamp: "Formatted at 2026-08-11 11:53:05 on dist-test-slave-bndk"
server_key: "05657cc35ad863404415f85d762e2cc1"
server_key_iv: "03e9aaac1fb55650fcd1afbd8330ee25"
server_key_version: "encryptionkey@0"
I20260811 11:53:05.695147 26902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:53:05.726150 26902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:53:05.730069 26902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:53:05.730260 26902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:53:05.730885 26902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:53:05.731822 26902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:53:05.731870 26902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:05.731914 26902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:53:05.731943 26902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:05.741547 26902 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:37931
I20260811 11:53:05.741582 27114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:37931 every 8 connection(s)
I20260811 11:53:05.742547 26902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:53:05.747150 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 26902
I20260811 11:53:05.747277 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:53:05.747617 3957 external_mini_cluster.cc:1428] Setting key 2f4f56e970f2496a6e3fd2775c0406eb
Aug 11 11:53:05 dist-test-slave-bndk krb5kdc[26279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449185, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:53:05.757028 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:05.744521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:49321 (local address 127.3.221.126:40419)
0811 11:53:05.744663 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:05.744667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:05.745797 (+ 1130us) server_negotiation.cc:407] Connection header received
0811 11:53:05.746703 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:05.746706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:05.746762 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:05.746866 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:05.748139 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.748794 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.750404 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.750636 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.750720 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:05.753351 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:05.753354 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:05.753356 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:05.753383 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:05.754734 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:05.755282 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:05.755285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:05.755286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:05.755396 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:05.755810 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:05.755813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:05.755815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:05.756035 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:05.756128 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:05.756755 (+ 627us) server_negotiation.cc:299] Negotiation successful
0811 11:53:05.756881 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 11:53:05.757861 27118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:05.744840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.67:49321)
0811 11:53:05.745626 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:05.745678 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:05.746485 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:05.747003 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:05.747010 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:05.747423 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:53:05.747980 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:05.747997 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.748956 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.748960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:05.750230 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:05.750243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.750796 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.750799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:05.750980 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:05.751574 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:53:05.751592 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:53:05.753110 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:53:05.754864 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:05.754870 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:05.754885 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:05.755168 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:05.755509 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:05.755512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:05.755514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:05.755680 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:05.756161 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:53:05.756167 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:53:05.756460 (+ 293us) client_negotiation.cc:769] Sending connection context
0811 11:53:05.756652 (+ 192us) client_negotiation.cc:240] Negotiation successful
0811 11:53:05.756945 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":130,"threads_started":1}
I20260811 11:53:05.759167 27115 heartbeater.cc:347] Connected to a master server at 127.3.221.126:40419
I20260811 11:53:05.759470 27115 heartbeater.cc:464] Registering TS with master...
I20260811 11:53:05.760023 27115 heartbeater.cc:511] Master 127.3.221.126:40419 requested a full tablet report, sending...
I20260811 11:53:05.761144 26359 ts_manager.cc:194] Registered new tserver with Master: 0efb25eba2184dbfb0e476a285fef172 (127.3.221.67:37931)
I20260811 11:53:05.761838 26359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:49321
I20260811 11:53:05.761945 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:53:05.772820 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:05.764216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58794 (local address 127.3.221.126:40419)
0811 11:53:05.764376 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:05.764380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:05.764625 (+ 245us) server_negotiation.cc:407] Connection header received
0811 11:53:05.764759 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:05.764761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:05.764804 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:05.764881 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:05.765701 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.766233 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.767151 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.767399 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.767486 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:05.768931 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:05.768934 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:05.768936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:05.768967 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:05.770769 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:05.771387 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:05.771390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:05.771392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:05.771481 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:05.771843 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:05.771846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:05.771847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:05.772121 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:05.772240 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:05.772527 (+ 287us) server_negotiation.cc:299] Negotiation successful
0811 11:53:05.772642 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 11:53:05.776397 26359 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:53:05.779775 26359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:53:05.797649 27140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:05.791516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.3.221.66:41217)
0811 11:53:05.792470 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:05.792483 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:05.792706 (+ 223us) server_negotiation.cc:407] Connection header received
0811 11:53:05.792854 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:05.792860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:05.793046 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:05.793181 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:53:05.793440 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.794262 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.796092 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.796837 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.796949 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:05.797175 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:05.797347 (+ 172us) server_negotiation.cc:299] Negotiation successful
0811 11:53:05.797475 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":849,"thread_start_us":139,"threads_started":1}
I20260811 11:53:05.797683 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:05.791197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:41217 (local address 127.0.0.1:50438)
0811 11:53:05.792601 (+ 1404us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:05.792615 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:05.792711 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:05.793172 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:05.793175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:05.793193 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:53:05.793332 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:05.793340 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.794385 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.794388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:05.795928 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:05.795935 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.796953 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.796957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:05.797086 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:05.797128 (+ 42us) client_negotiation.cc:769] Sending connection context
0811 11:53:05.797312 (+ 184us) client_negotiation.cc:240] Negotiation successful
0811 11:53:05.797544 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1340,"mutex_wait_us":139,"thread_start_us":213,"threads_started":1}
I20260811 11:53:05.798079 27139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:05.791052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:37931 (local address 127.0.0.1:42390)
0811 11:53:05.791951 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:05.792008 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:05.792141 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:05.792881 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:05.792888 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:05.792926 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:53:05.793091 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:05.793100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.794572 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.794575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:05.795916 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:05.795928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.797149 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.797153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:05.797282 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:05.797310 (+ 28us) client_negotiation.cc:769] Sending connection context
0811 11:53:05.797363 (+ 53us) client_negotiation.cc:240] Negotiation successful
0811 11:53:05.797511 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":42,"thread_start_us":92,"threads_started":1}
I20260811 11:53:05.798125 27141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:05.791787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42390 (local address 127.3.221.67:37931)
0811 11:53:05.792197 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:05.792210 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:05.792237 (+ 27us) server_negotiation.cc:407] Connection header received
0811 11:53:05.792450 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:05.792458 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:05.792695 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:05.792899 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:53:05.793249 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.794438 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.796119 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.796996 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.797641 (+ 645us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:05.797797 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:05.797854 (+ 57us) server_negotiation.cc:299] Negotiation successful
0811 11:53:05.797969 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":99,"threads_started":1}
I20260811 11:53:05.798282 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:05.791481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:32839 (local address 127.0.0.1:34846)
0811 11:53:05.792765 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:05.792779 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:05.792880 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:05.793508 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:05.793511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:05.793530 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:53:05.793618 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:05.793624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.795140 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.795143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:05.796320 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:05.796333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.797900 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.797904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:05.798001 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:05.798024 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 11:53:05.798096 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 11:53:05.798168 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1221,"thread_start_us":106,"threads_started":1}
I20260811 11:53:05.798686 27142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:05.791940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34846 (local address 127.3.221.65:32839)
0811 11:53:05.792464 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:05.792471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:05.793073 (+ 602us) server_negotiation.cc:407] Connection header received
0811 11:53:05.793158 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:05.793163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:05.793380 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:05.793524 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:53:05.793727 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.794987 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.796449 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.797622 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.797721 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:05.797836 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:05.798471 (+ 635us) server_negotiation.cc:299] Negotiation successful
0811 11:53:05.798569 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":109,"threads_started":1}
I20260811 11:53:05.801712 26569 tablet_service.cc:1467] Processing CreateTablet for tablet bf92b17b99e641f18aaac341a6b5bc33 (DEFAULT_TABLE table=test-table [id=2f02531afe8242b0812db9921802df39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:53:05.802067 27010 tablet_service.cc:1467] Processing CreateTablet for tablet bf92b17b99e641f18aaac341a6b5bc33 (DEFAULT_TABLE table=test-table [id=2f02531afe8242b0812db9921802df39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:53:05.801853 26791 tablet_service.cc:1467] Processing CreateTablet for tablet bf92b17b99e641f18aaac341a6b5bc33 (DEFAULT_TABLE table=test-table [id=2f02531afe8242b0812db9921802df39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:53:05.802826 26569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf92b17b99e641f18aaac341a6b5bc33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:53:05.803122 26791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf92b17b99e641f18aaac341a6b5bc33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:53:05.803133 27010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf92b17b99e641f18aaac341a6b5bc33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:53:05.810220 27143 tablet_bootstrap.cc:492] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24: Bootstrap starting.
I20260811 11:53:05.811306 27144 tablet_bootstrap.cc:492] T bf92b17b99e641f18aaac341a6b5bc33 P 903b03d331eb406da01ced709713f1fe: Bootstrap starting.
I20260811 11:53:05.811478 27146 tablet_bootstrap.cc:492] T bf92b17b99e641f18aaac341a6b5bc33 P 0efb25eba2184dbfb0e476a285fef172: Bootstrap starting.
I20260811 11:53:05.813200 27143 tablet_bootstrap.cc:654] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24: Neither blocks nor log segments found. Creating new log.
I20260811 11:53:05.814023 27144 tablet_bootstrap.cc:654] T bf92b17b99e641f18aaac341a6b5bc33 P 903b03d331eb406da01ced709713f1fe: Neither blocks nor log segments found. Creating new log.
I20260811 11:53:05.814038 27146 tablet_bootstrap.cc:654] T bf92b17b99e641f18aaac341a6b5bc33 P 0efb25eba2184dbfb0e476a285fef172: Neither blocks nor log segments found. Creating new log.
I20260811 11:53:05.814319 27143 log.cc:824] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24: Log is configured to *not* fsync() on all Append() calls
I20260811 11:53:05.814925 27144 log.cc:824] T bf92b17b99e641f18aaac341a6b5bc33 P 903b03d331eb406da01ced709713f1fe: Log is configured to *not* fsync() on all Append() calls
I20260811 11:53:05.814925 27146 log.cc:824] T bf92b17b99e641f18aaac341a6b5bc33 P 0efb25eba2184dbfb0e476a285fef172: Log is configured to *not* fsync() on all Append() calls
I20260811 11:53:05.816766 27143 tablet_bootstrap.cc:492] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24: No bootstrap required, opened a new log
I20260811 11:53:05.816973 27146 tablet_bootstrap.cc:492] T bf92b17b99e641f18aaac341a6b5bc33 P 0efb25eba2184dbfb0e476a285fef172: No bootstrap required, opened a new log
I20260811 11:53:05.816967 27144 tablet_bootstrap.cc:492] T bf92b17b99e641f18aaac341a6b5bc33 P 903b03d331eb406da01ced709713f1fe: No bootstrap required, opened a new log
I20260811 11:53:05.816994 27143 ts_tablet_manager.cc:1397] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20260811 11:53:05.817168 27144 ts_tablet_manager.cc:1397] T bf92b17b99e641f18aaac341a6b5bc33 P 903b03d331eb406da01ced709713f1fe: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 11:53:05.817168 27146 ts_tablet_manager.cc:1397] T bf92b17b99e641f18aaac341a6b5bc33 P 0efb25eba2184dbfb0e476a285fef172: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 11:53:05.820668 27143 raft_consensus.cc:348] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:05.820950 27143 raft_consensus.cc:374] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:53:05.821007 27143 raft_consensus.cc:729] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc6b1d1884454450ae1f9cbc66519a24, State: Initialized, Role: FOLLOWER
I20260811 11:53:05.821084 27144 raft_consensus.cc:348] T bf92b17b99e641f18aaac341a6b5bc33 P 903b03d331eb406da01ced709713f1fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:05.821084 27146 raft_consensus.cc:348] T bf92b17b99e641f18aaac341a6b5bc33 P 0efb25eba2184dbfb0e476a285fef172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:05.821413 27146 raft_consensus.cc:374] T bf92b17b99e641f18aaac341a6b5bc33 P 0efb25eba2184dbfb0e476a285fef172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:53:05.821413 27144 raft_consensus.cc:374] T bf92b17b99e641f18aaac341a6b5bc33 P 903b03d331eb406da01ced709713f1fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:53:05.821475 27144 raft_consensus.cc:729] T bf92b17b99e641f18aaac341a6b5bc33 P 903b03d331eb406da01ced709713f1fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903b03d331eb406da01ced709713f1fe, State: Initialized, Role: FOLLOWER
I20260811 11:53:05.821487 27146 raft_consensus.cc:729] T bf92b17b99e641f18aaac341a6b5bc33 P 0efb25eba2184dbfb0e476a285fef172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0efb25eba2184dbfb0e476a285fef172, State: Initialized, Role: FOLLOWER
I20260811 11:53:05.821589 27143 consensus_queue.cc:260] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:05.821916 27146 consensus_queue.cc:260] T bf92b17b99e641f18aaac341a6b5bc33 P 0efb25eba2184dbfb0e476a285fef172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:05.821916 27144 consensus_queue.cc:260] T bf92b17b99e641f18aaac341a6b5bc33 P 903b03d331eb406da01ced709713f1fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:05.822745 27143 ts_tablet_manager.cc:1428] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260811 11:53:05.822883 27144 ts_tablet_manager.cc:1428] T bf92b17b99e641f18aaac341a6b5bc33 P 903b03d331eb406da01ced709713f1fe: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260811 11:53:05.822883 27146 ts_tablet_manager.cc:1428] T bf92b17b99e641f18aaac341a6b5bc33 P 0efb25eba2184dbfb0e476a285fef172: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260811 11:53:05.823132 26679 heartbeater.cc:503] Master 127.3.221.126:40419 was elected leader, sending a full tablet report...
I20260811 11:53:05.823212 27115 heartbeater.cc:503] Master 127.3.221.126:40419 was elected leader, sending a full tablet report...
I20260811 11:53:05.824246 26894 heartbeater.cc:503] Master 127.3.221.126:40419 was elected leader, sending a full tablet report...
I20260811 11:53:05.878412 27150 raft_consensus.cc:482] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:53:05.878664 27150 raft_consensus.cc:504] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:05.880039 27150 leader_election.cc:288] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0efb25eba2184dbfb0e476a285fef172 (127.3.221.67:37931), 903b03d331eb406da01ced709713f1fe (127.3.221.66:41217)
I20260811 11:53:05.884620 27141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:05.880579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:33061 (local address 127.3.221.67:37931)
0811 11:53:05.880759 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:05.880763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:05.880954 (+ 191us) server_negotiation.cc:407] Connection header received
0811 11:53:05.881112 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:05.881115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:05.881168 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:05.881292 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:53:05.881607 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.882435 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.883658 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.884073 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.884186 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:05.884241 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:05.884411 (+ 170us) server_negotiation.cc:299] Negotiation successful
0811 11:53:05.884490 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260811 11:53:05.884641 27140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:05.880456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:60283 (local address 127.3.221.66:41217)
0811 11:53:05.880583 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:05.880588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:05.880995 (+ 407us) server_negotiation.cc:407] Connection header received
0811 11:53:05.881197 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:05.881200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:05.881275 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:05.881346 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:53:05.881683 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.882433 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.883645 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.884059 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.884177 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:05.884235 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:05.884450 (+ 215us) server_negotiation.cc:299] Negotiation successful
0811 11:53:05.884526 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260811 11:53:05.884661 27158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:05.880338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:41217 (local address 127.3.221.65:60283)
0811 11:53:05.880917 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:05.880928 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:05.881051 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:05.881425 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:05.881428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:05.881448 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:53:05.881565 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:05.881571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.882568 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.882571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:05.883494 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:05.883506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.884198 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.884201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:05.884319 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:05.884338 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 11:53:05.884415 (+ 77us) client_negotiation.cc:240] Negotiation successful
0811 11:53:05.884486 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":227,"thread_start_us":124,"threads_started":1}
I20260811 11:53:05.884665 27155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:05.880339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:37931 (local address 127.3.221.65:33061)
0811 11:53:05.880833 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:05.880849 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:05.881000 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:05.881304 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:05.881308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:05.881326 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:53:05.881488 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:05.881495 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.882597 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.882600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:05.883495 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:05.883505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.884188 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.884192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:05.884288 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:05.884310 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 11:53:05.884372 (+ 62us) client_negotiation.cc:240] Negotiation successful
0811 11:53:05.884460 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":10,"thread_start_us":160,"threads_started":1}
I20260811 11:53:05.885509 27042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf92b17b99e641f18aaac341a6b5bc33" candidate_uuid: "bc6b1d1884454450ae1f9cbc66519a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0efb25eba2184dbfb0e476a285fef172" is_pre_election: true
I20260811 11:53:05.885509 26822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf92b17b99e641f18aaac341a6b5bc33" candidate_uuid: "bc6b1d1884454450ae1f9cbc66519a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903b03d331eb406da01ced709713f1fe" is_pre_election: true
I20260811 11:53:05.885859 26822 raft_consensus.cc:2454] T bf92b17b99e641f18aaac341a6b5bc33 P 903b03d331eb406da01ced709713f1fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc6b1d1884454450ae1f9cbc66519a24 in term 0.
I20260811 11:53:05.885860 27042 raft_consensus.cc:2454] T bf92b17b99e641f18aaac341a6b5bc33 P 0efb25eba2184dbfb0e476a285fef172 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc6b1d1884454450ae1f9cbc66519a24 in term 0.
I20260811 11:53:05.886431 26495 leader_election.cc:302] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 903b03d331eb406da01ced709713f1fe, bc6b1d1884454450ae1f9cbc66519a24; no voters:
I20260811 11:53:05.886818 27150 raft_consensus.cc:2781] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:53:05.887002 27150 raft_consensus.cc:482] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:53:05.887042 27150 raft_consensus.cc:3037] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:53:05.888456 27150 raft_consensus.cc:504] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:05.888866 27150 leader_election.cc:288] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 1 election: Requested vote from peers 0efb25eba2184dbfb0e476a285fef172 (127.3.221.67:37931), 903b03d331eb406da01ced709713f1fe (127.3.221.66:41217)
I20260811 11:53:05.889334 27042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf92b17b99e641f18aaac341a6b5bc33" candidate_uuid: "bc6b1d1884454450ae1f9cbc66519a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0efb25eba2184dbfb0e476a285fef172"
I20260811 11:53:05.889329 26822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf92b17b99e641f18aaac341a6b5bc33" candidate_uuid: "bc6b1d1884454450ae1f9cbc66519a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903b03d331eb406da01ced709713f1fe"
I20260811 11:53:05.889473 26822 raft_consensus.cc:3037] T bf92b17b99e641f18aaac341a6b5bc33 P 903b03d331eb406da01ced709713f1fe [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:53:05.889465 27042 raft_consensus.cc:3037] T bf92b17b99e641f18aaac341a6b5bc33 P 0efb25eba2184dbfb0e476a285fef172 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:53:05.890894 26822 raft_consensus.cc:2454] T bf92b17b99e641f18aaac341a6b5bc33 P 903b03d331eb406da01ced709713f1fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc6b1d1884454450ae1f9cbc66519a24 in term 1.
I20260811 11:53:05.890962 27042 raft_consensus.cc:2454] T bf92b17b99e641f18aaac341a6b5bc33 P 0efb25eba2184dbfb0e476a285fef172 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc6b1d1884454450ae1f9cbc66519a24 in term 1.
I20260811 11:53:05.891347 26496 leader_election.cc:302] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0efb25eba2184dbfb0e476a285fef172, bc6b1d1884454450ae1f9cbc66519a24; no voters:
I20260811 11:53:05.891559 27150 raft_consensus.cc:2781] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:53:05.892046 27150 raft_consensus.cc:686] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [term 1 LEADER]: Becoming Leader. State: Replica: bc6b1d1884454450ae1f9cbc66519a24, State: Running, Role: LEADER
I20260811 11:53:05.892436 27150 consensus_queue.cc:237] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:05.895907 26351 catalog_manager.cc:5495] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc6b1d1884454450ae1f9cbc66519a24 (127.3.221.65). New cstate: current_term: 1 leader_uuid: "bc6b1d1884454450ae1f9cbc66519a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } health_report { overall_health: HEALTHY } } }
I20260811 11:53:05.925324 27142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:05.921565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34860 (local address 127.3.221.65:32839)
0811 11:53:05.921686 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:05.921690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:05.921705 (+ 15us) server_negotiation.cc:407] Connection header received
0811 11:53:05.921808 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:05.921810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:05.921866 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:05.921968 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:53:05.922277 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.922841 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.923749 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:05.923980 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:05.924072 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:05.924418 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:53:05.924882 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:53:05.924983 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:05.925098 (+ 115us) server_negotiation.cc:299] Negotiation successful
0811 11:53:05.925154 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 11:53:05.931665 26822 raft_consensus.cc:1261] T bf92b17b99e641f18aaac341a6b5bc33 P 903b03d331eb406da01ced709713f1fe [term 1 FOLLOWER]: Refusing update from remote peer bc6b1d1884454450ae1f9cbc66519a24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:53:05.931665 27042 raft_consensus.cc:1261] T bf92b17b99e641f18aaac341a6b5bc33 P 0efb25eba2184dbfb0e476a285fef172 [term 1 FOLLOWER]: Refusing update from remote peer bc6b1d1884454450ae1f9cbc66519a24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:53:05.932472 27163 consensus_queue.cc:1035] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:53:05.932675 27150 consensus_queue.cc:1035] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:53:05.943305 27171 mvcc.cc:205] Tried to move back new op lower bound from 7317295865568260096 to 7317295865418391552. Current Snapshot: MvccSnapshot[applied={T|T < 7317295865568260096}]
I20260811 11:53:05.950106 26351 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:53:05.950291 26351 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:53:05.952898 26351 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82: Sending DeleteTablet for 3 replicas of tablet bf92b17b99e641f18aaac341a6b5bc33
I20260811 11:53:05.953889 27010 tablet_service.cc:1514] Processing DeleteTablet for tablet bf92b17b99e641f18aaac341a6b5bc33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:53:05 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:42390
I20260811 11:53:05.953892 26791 tablet_service.cc:1514] Processing DeleteTablet for tablet bf92b17b99e641f18aaac341a6b5bc33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:53:05 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:50438
I20260811 11:53:05.954284 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 26461
I20260811 11:53:05.954209 26569 tablet_service.cc:1514] Processing DeleteTablet for tablet bf92b17b99e641f18aaac341a6b5bc33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:53:05 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:34846
I20260811 11:53:05.962597 27183 tablet_replica.cc:330] T bf92b17b99e641f18aaac341a6b5bc33 P 903b03d331eb406da01ced709713f1fe: stopping tablet replica
I20260811 11:53:05.962989 27182 tablet_replica.cc:330] T bf92b17b99e641f18aaac341a6b5bc33 P 0efb25eba2184dbfb0e476a285fef172: stopping tablet replica
I20260811 11:53:05.963766 27183 raft_consensus.cc:2229] T bf92b17b99e641f18aaac341a6b5bc33 P 903b03d331eb406da01ced709713f1fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:53:05.964138 27182 raft_consensus.cc:2229] T bf92b17b99e641f18aaac341a6b5bc33 P 0efb25eba2184dbfb0e476a285fef172 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:53:05.964668 27182 raft_consensus.cc:2258] T bf92b17b99e641f18aaac341a6b5bc33 P 0efb25eba2184dbfb0e476a285fef172 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:53:05.965140 27183 raft_consensus.cc:2258] T bf92b17b99e641f18aaac341a6b5bc33 P 903b03d331eb406da01ced709713f1fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:53:05.975140 27182 ts_tablet_manager.cc:1905] T bf92b17b99e641f18aaac341a6b5bc33 P 0efb25eba2184dbfb0e476a285fef172: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:53:05.975335 27183 ts_tablet_manager.cc:1905] T bf92b17b99e641f18aaac341a6b5bc33 P 903b03d331eb406da01ced709713f1fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:53:05.977931 27183 ts_tablet_manager.cc:1918] T bf92b17b99e641f18aaac341a6b5bc33 P 903b03d331eb406da01ced709713f1fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:53:05.978062 27183 log.cc:1196] T bf92b17b99e641f18aaac341a6b5bc33 P 903b03d331eb406da01ced709713f1fe: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/bf92b17b99e641f18aaac341a6b5bc33
I20260811 11:53:05.978062 27182 ts_tablet_manager.cc:1918] T bf92b17b99e641f18aaac341a6b5bc33 P 0efb25eba2184dbfb0e476a285fef172: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:53:05.978163 27182 log.cc:1196] T bf92b17b99e641f18aaac341a6b5bc33 P 0efb25eba2184dbfb0e476a285fef172: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/bf92b17b99e641f18aaac341a6b5bc33
I20260811 11:53:05.978415 27183 ts_tablet_manager.cc:1939] T bf92b17b99e641f18aaac341a6b5bc33 P 903b03d331eb406da01ced709713f1fe: Deleting consensus metadata
I20260811 11:53:05.978577 27182 ts_tablet_manager.cc:1939] T bf92b17b99e641f18aaac341a6b5bc33 P 0efb25eba2184dbfb0e476a285fef172: Deleting consensus metadata
I20260811 11:53:05.979825 26338 catalog_manager.cc:4841] TS 0efb25eba2184dbfb0e476a285fef172 (127.3.221.67:37931): tablet bf92b17b99e641f18aaac341a6b5bc33 (table test-table [id=2f02531afe8242b0812db9921802df39]) successfully deleted
I20260811 11:53:05.979825 26337 catalog_manager.cc:4841] TS 903b03d331eb406da01ced709713f1fe (127.3.221.66:41217): tablet bf92b17b99e641f18aaac341a6b5bc33 (table test-table [id=2f02531afe8242b0812db9921802df39]) successfully deleted
I20260811 11:53:05.983940 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 26687
W20260811 11:53:05.983944 26328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.221.65:32839 (error 108)
I20260811 11:53:05.993445 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:05.992830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:32839 (local address 127.0.0.1:34870)
0811 11:53:05.993172 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:05.993303 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.65:32839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243}
W20260811 11:53:05.994014 26328 catalog_manager.cc:4584] TS bc6b1d1884454450ae1f9cbc66519a24 (127.3.221.65:32839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf92b17b99e641f18aaac341a6b5bc33: Network error: Client connection negotiation failed: client connection to 127.3.221.65:32839: connect: Connection refused (error 111)
I20260811 11:53:06.002801 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 26902
I20260811 11:53:06.023586 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 26304
I20260811 11:53:06.037853 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:40419
--webserver_interface=127.3.221.126
--webserver_port=43831
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:40419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:53:06.170058 27187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:53:06.170373 27187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:53:06.170415 27187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:53:06.174966 27187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:53:06.175061 27187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:53:06.175088 27187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:53:06.175104 27187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:53:06.175119 27187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:53:06.181324 27187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:40419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:40419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=43831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:53:06.182574 27187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:53:06.183863 27187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:53:06.199337 27200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:06.199494 27197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:06.199539 27195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:06.200125 27187 server_base.cc:1034] running on GCE node
I20260811 11:53:06.201164 27187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:53:06.202638 27187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:53:06.203847 27187 hybrid_clock.cc:648] HybridClock initialized: now 1786449186203822 us; error 37 us; skew 500 ppm
Aug 11 11:53:06 dist-test-slave-bndk krb5kdc[26279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449186, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:53:06.207885 27187 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:53:06.209638 27187 webserver.cc:460] Webserver started at http://127.3.221.126:43831/ using document root <none> and password file <none>
I20260811 11:53:06.210739 27187 fs_manager.cc:359] Metadata directory not provided
I20260811 11:53:06.210889 27187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:53:06.216965 27187 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.003s
I20260811 11:53:06.220017 27215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:06.221885 27187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:53:06.222043 27187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "691ca47614cf4870a72b077bdc797e82"
format_stamp: "Formatted at 2026-08-11 11:53:04 on dist-test-slave-bndk"
server_key: "830487391d2e4b55a211fd099941d060"
server_key_iv: "c8ef99263eaba4e050d7ad6383ad42ee"
server_key_version: "encryptionkey@0"
I20260811 11:53:06.222756 27187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:53:06.245651 27187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:53:06.249347 27187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:53:06.249553 27187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:53:06.258086 27187 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:40419
I20260811 11:53:06.258385 27302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:40419 every 8 connection(s)
I20260811 11:53:06.259402 27187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:53:06.264425 27303 sys_catalog.cc:263] Verifying existing consensus state
I20260811 11:53:06.266390 27303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82: Bootstrap starting.
I20260811 11:53:06.267375 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 27187
I20260811 11:53:06.267866 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:32839
--local_ip_for_outbound_sockets=127.3.221.65
--tserver_master_addrs=127.3.221.126:40419
--webserver_port=42685
--webserver_interface=127.3.221.65
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:53:06.286031 27303 log.cc:824] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82: Log is configured to *not* fsync() on all Append() calls
I20260811 11:53:06.296382 27303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:53:06.296818 27303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82: Bootstrap complete.
I20260811 11:53:06.301328 27303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691ca47614cf4870a72b077bdc797e82" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } }
I20260811 11:53:06.301787 27303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691ca47614cf4870a72b077bdc797e82, State: Initialized, Role: FOLLOWER
I20260811 11:53:06.302333 27303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691ca47614cf4870a72b077bdc797e82" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } }
I20260811 11:53:06.302596 27303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:53:06.302688 27303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:53:06.302790 27303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 1 FOLLOWER]: Advancing to term 2
I20260811 11:53:06.305136 27303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691ca47614cf4870a72b077bdc797e82" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } }
I20260811 11:53:06.305651 27303 leader_election.cc:302] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 691ca47614cf4870a72b077bdc797e82; no voters:
I20260811 11:53:06.306017 27303 leader_election.cc:288] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 11:53:06.306130 27311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 11:53:06.306327 27311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 2 LEADER]: Becoming Leader. State: Replica: 691ca47614cf4870a72b077bdc797e82, State: Running, Role: LEADER
I20260811 11:53:06.306650 27311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691ca47614cf4870a72b077bdc797e82" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } }
I20260811 11:53:06.307395 27303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:53:06.307929 27311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 691ca47614cf4870a72b077bdc797e82. Latest consensus state: current_term: 2 leader_uuid: "691ca47614cf4870a72b077bdc797e82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691ca47614cf4870a72b077bdc797e82" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } } }
I20260811 11:53:06.308064 27311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [sys.catalog]: This master's current role is: LEADER
I20260811 11:53:06.308360 27315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "691ca47614cf4870a72b077bdc797e82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691ca47614cf4870a72b077bdc797e82" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } } }
I20260811 11:53:06.308477 27319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:53:06.308550 27315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [sys.catalog]: This master's current role is: LEADER
I20260811 11:53:06.314060 27319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:53:06.314872 27319 catalog_manager.cc:1259] Loaded cluster ID: 94cac514bcb94264a4c93de5648b35ba
I20260811 11:53:06.314942 27319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:53:06.316478 27319 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260811 11:53:06.321285 27319 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260811 11:53:06.327445 27319 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:53:06.328365 27319 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82: Loaded TSK: 0
I20260811 11:53:06.329702 27319 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 11:53:06.397022 27307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:53:06.397396 27307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:53:06.397488 27307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:53:06.402005 27307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:53:06.402101 27307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:53:06.402191 27307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:53:06.408164 27307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:32839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=42685
--tserver_master_addrs=127.3.221.126:40419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:53:06.409435 27307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:53:06.410491 27307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:53:06.422883 27348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:06.422940 27350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:06.423491 27353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:06.423890 27307 server_base.cc:1034] running on GCE node
I20260811 11:53:06.424424 27307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:53:06.425356 27307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:53:06.426597 27307 hybrid_clock.cc:648] HybridClock initialized: now 1786449186426565 us; error 44 us; skew 500 ppm
Aug 11 11:53:06 dist-test-slave-bndk krb5kdc[26279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449186, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:53:06.430135 27307 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:53:06.431381 27307 webserver.cc:460] Webserver started at http://127.3.221.65:42685/ using document root <none> and password file <none>
I20260811 11:53:06.432029 27307 fs_manager.cc:359] Metadata directory not provided
I20260811 11:53:06.432126 27307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:53:06.436594 27307 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260811 11:53:06.439011 27364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:06.440415 27307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 11:53:06.440541 27307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "bc6b1d1884454450ae1f9cbc66519a24"
format_stamp: "Formatted at 2026-08-11 11:53:05 on dist-test-slave-bndk"
server_key: "c92c5d7a16739b8ed3323f88351d5264"
server_key_iv: "310a2536254248c76216b2c8eb1e99ed"
server_key_version: "encryptionkey@0"
I20260811 11:53:06.441150 27307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:53:06.462553 27307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:53:06.466238 27307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:53:06.466451 27307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:53:06.467110 27307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:53:06.468390 27375 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260811 11:53:06.474357 27307 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 11:53:06.474462 27307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260811 11:53:06.474649 27307 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 11:53:06.477054 27307 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 11:53:06.477123 27307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260811 11:53:06.477483 27375 tablet_bootstrap.cc:492] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24: Bootstrap starting.
I20260811 11:53:06.488088 27307 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:32839
I20260811 11:53:06.488169 27545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:32839 every 8 connection(s)
I20260811 11:53:06.489192 27307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:53:06.497093 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 27307
I20260811 11:53:06.497648 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:41217
--local_ip_for_outbound_sockets=127.3.221.66
--tserver_master_addrs=127.3.221.126:40419
--webserver_port=34647
--webserver_interface=127.3.221.66
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:53:06.498309 27375 log.cc:824] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24: Log is configured to *not* fsync() on all Append() calls
Aug 11 11:53:06 dist-test-slave-bndk krb5kdc[26279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449186, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:53:06.507292 27375 tablet_bootstrap.cc:492] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:53:06.507691 27375 tablet_bootstrap.cc:492] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24: Bootstrap complete.
I20260811 11:53:06.509188 27375 ts_tablet_manager.cc:1397] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24: Time spent bootstrapping tablet: real 0.032s user 0.020s sys 0.011s
I20260811 11:53:06.510100 27549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:06.491529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.65:33851)
0811 11:53:06.492198 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:06.492266 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:06.493161 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:06.494476 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:06.494485 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:06.494970 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:53:06.495465 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:06.495481 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.496861 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.496864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:06.497944 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:06.497958 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.498916 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.498920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:06.499112 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:06.501423 (+ 2311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:53:06.501446 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:53:06.503398 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:53:06.506129 (+ 2731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:06.506136 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:06.506149 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:06.506521 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:06.506994 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:06.506997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:06.506999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:06.507187 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:06.508070 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:53:06.508077 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:53:06.508402 (+ 325us) client_negotiation.cc:769] Sending connection context
0811 11:53:06.508601 (+ 199us) client_negotiation.cc:240] Negotiation successful
0811 11:53:06.508959 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":183,"threads_started":1}
I20260811 11:53:06.510234 27551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:06.491752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:33851 (local address 127.3.221.126:40419)
0811 11:53:06.492294 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:06.492314 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:06.492398 (+ 84us) server_negotiation.cc:407] Connection header received
0811 11:53:06.493668 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:06.493680 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:06.494024 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:06.494378 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:06.495660 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.496679 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.498078 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.498624 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.498962 (+ 338us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:06.503679 (+ 4717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:06.503687 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:06.503707 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:06.503749 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:06.505969 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:06.506676 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:06.506684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:06.506690 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:06.506864 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:06.507299 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:06.507302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:06.507304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:06.507848 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:06.508097 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:06.508831 (+ 734us) server_negotiation.cc:299] Negotiation successful
0811 11:53:06.509314 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":148,"threads_started":1}
I20260811 11:53:06.512306 27375 raft_consensus.cc:348] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:06.512749 27375 raft_consensus.cc:729] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc6b1d1884454450ae1f9cbc66519a24, State: Initialized, Role: FOLLOWER
I20260811 11:53:06.513309 27375 consensus_queue.cc:260] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:06.513696 27547 heartbeater.cc:347] Connected to a master server at 127.3.221.126:40419
I20260811 11:53:06.513998 27547 heartbeater.cc:464] Registering TS with master...
I20260811 11:53:06.514353 27375 ts_tablet_manager.cc:1428] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260811 11:53:06.514611 27547 heartbeater.cc:511] Master 127.3.221.126:40419 requested a full tablet report, sending...
I20260811 11:53:06.517663 27243 ts_manager.cc:194] Registered new tserver with Master: bc6b1d1884454450ae1f9cbc66519a24 (127.3.221.65:32839)
I20260811 11:53:06.521540 27243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:33851
I20260811 11:53:06.524647 27547 heartbeater.cc:503] Master 127.3.221.126:40419 was elected leader, sending a full tablet report...
Aug 11 11:53:06 dist-test-slave-bndk krb5kdc[26279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449186, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for kudu/127.3.221.65@KRBTEST.COM
I20260811 11:53:06.532956 27555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:06.520888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:32839 (local address 127.0.0.1:34878)
0811 11:53:06.522031 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:06.522079 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:06.522230 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:06.522875 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:06.522882 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:06.523309 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:53:06.523428 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:06.523435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.524409 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.524412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:06.525108 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:06.525121 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.525977 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.525982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:06.526103 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:06.526794 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:53:06.526819 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:53:06.528518 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:53:06.530893 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:06.530903 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:06.530908 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:06.531170 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:06.531691 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:06.531694 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:06.531696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:06.531794 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:06.532368 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:53:06.532374 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:53:06.532511 (+ 137us) client_negotiation.cc:769] Sending connection context
0811 11:53:06.532627 (+ 116us) client_negotiation.cc:240] Negotiation successful
0811 11:53:06.532808 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"thread_start_us":157,"threads_started":1}
I20260811 11:53:06.533540 27558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:06.521027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34878 (local address 127.3.221.65:32839)
0811 11:53:06.521634 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:06.521650 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:06.522377 (+ 727us) server_negotiation.cc:407] Connection header received
0811 11:53:06.522464 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:06.522470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:06.522687 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:06.522858 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:06.523554 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.524285 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.525294 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.525802 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.525948 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:06.528681 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:06.528693 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:06.528709 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:06.528740 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:06.530733 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:06.531293 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:06.531300 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:06.531301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:06.531556 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:06.531902 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:06.531906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:06.531908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:06.532252 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:06.532389 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:06.532736 (+ 347us) server_negotiation.cc:299] Negotiation successful
0811 11:53:06.532966 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":206,"threads_started":1}
I20260811 11:53:06.534664 27441 tablet_service.cc:1514] Processing DeleteTablet for tablet bf92b17b99e641f18aaac341a6b5bc33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:53:05 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:34878
I20260811 11:53:06.535594 27559 tablet_replica.cc:330] stopping tablet replica
I20260811 11:53:06.535974 27559 raft_consensus.cc:2229] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:53:06.536265 27559 raft_consensus.cc:2258] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:53:06.537602 27559 ts_tablet_manager.cc:1905] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:53:06.542655 27559 ts_tablet_manager.cc:1918] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:53:06.542805 27559 log.cc:1196] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/wal/wals/bf92b17b99e641f18aaac341a6b5bc33
I20260811 11:53:06.543097 27559 ts_tablet_manager.cc:1939] T bf92b17b99e641f18aaac341a6b5bc33 P bc6b1d1884454450ae1f9cbc66519a24: Deleting consensus metadata
I20260811 11:53:06.545621 27218 catalog_manager.cc:4841] TS bc6b1d1884454450ae1f9cbc66519a24 (127.3.221.65:32839): tablet bf92b17b99e641f18aaac341a6b5bc33 (table test-table [id=2f02531afe8242b0812db9921802df39]) successfully deleted
W20260811 11:53:06.640069 27553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:53:06.640389 27553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:53:06.640443 27553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:53:06.644649 27553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:53:06.644727 27553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:53:06.644788 27553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:53:06.650496 27553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:41217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=34647
--tserver_master_addrs=127.3.221.126:40419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:53:06.651825 27553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:53:06.652925 27553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:53:06.663100 27573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:06.663307 27568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:06.663374 27565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:06.663488 27553 server_base.cc:1034] running on GCE node
I20260811 11:53:06.663990 27553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:53:06.664695 27553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:53:06.665956 27553 hybrid_clock.cc:648] HybridClock initialized: now 1786449186665934 us; error 38 us; skew 500 ppm
Aug 11 11:53:06 dist-test-slave-bndk krb5kdc[26279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449186, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:53:06.669063 27553 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:53:06.670084 27553 webserver.cc:460] Webserver started at http://127.3.221.66:34647/ using document root <none> and password file <none>
I20260811 11:53:06.670838 27553 fs_manager.cc:359] Metadata directory not provided
I20260811 11:53:06.670953 27553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:53:06.675800 27553 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 11:53:06.678982 27584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:06.680692 27553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 11:53:06.680846 27553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "903b03d331eb406da01ced709713f1fe"
format_stamp: "Formatted at 2026-08-11 11:53:05 on dist-test-slave-bndk"
server_key: "65bd0c4a5c31add6f734b499f31f9aba"
server_key_iv: "ddcab6a90e1c315d970cc78d183a9f6a"
server_key_version: "encryptionkey@0"
I20260811 11:53:06.681618 27553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:53:06.695307 27553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:53:06.698805 27553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:53:06.699020 27553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:53:06.699827 27553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:53:06.700881 27553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:53:06.700932 27553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:06.700990 27553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:53:06.701020 27553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:06.710503 27553 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:41217
I20260811 11:53:06.710528 27769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:41217 every 8 connection(s)
I20260811 11:53:06.711625 27553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:53:06.717332 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 27553
I20260811 11:53:06.717835 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:37931
--local_ip_for_outbound_sockets=127.3.221.67
--tserver_master_addrs=127.3.221.126:40419
--webserver_port=45853
--webserver_interface=127.3.221.67
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 11:53:06 dist-test-slave-bndk krb5kdc[26279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449186, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:53:06.727478 27551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:06.713828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:43547 (local address 127.3.221.126:40419)
0811 11:53:06.713989 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:06.713994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:06.715736 (+ 1742us) server_negotiation.cc:407] Connection header received
0811 11:53:06.716810 (+ 1074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:06.716813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:06.716871 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:06.716952 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:06.718706 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.719267 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.720497 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.720739 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.720841 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:06.723706 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:06.723711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:06.723713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:06.723759 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:06.725168 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:06.725784 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:06.725787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:06.725788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:06.725873 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:06.726267 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:06.726269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:06.726271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:06.726474 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:06.726565 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:06.727142 (+ 577us) server_negotiation.cc:299] Negotiation successful
0811 11:53:06.727300 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 11:53:06.728389 27776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:06.714256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.66:43547)
0811 11:53:06.715470 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:06.715554 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:06.716540 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:06.717190 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:06.717200 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:06.717769 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:53:06.718502 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:06.718525 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.719430 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.719434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:06.720357 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:06.720369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.720908 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.720912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:06.721093 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:06.721725 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:53:06.721743 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:53:06.723498 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:53:06.725333 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:06.725341 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:06.725360 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:06.725671 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:06.725985 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:06.725988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:06.725990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:06.726159 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:06.726580 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:53:06.726588 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:53:06.726871 (+ 283us) client_negotiation.cc:769] Sending connection context
0811 11:53:06.727057 (+ 186us) client_negotiation.cc:240] Negotiation successful
0811 11:53:06.727445 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":536,"threads_started":1}
I20260811 11:53:06.729708 27770 heartbeater.cc:347] Connected to a master server at 127.3.221.126:40419
I20260811 11:53:06.730010 27770 heartbeater.cc:464] Registering TS with master...
I20260811 11:53:06.730631 27770 heartbeater.cc:511] Master 127.3.221.126:40419 requested a full tablet report, sending...
I20260811 11:53:06.732170 27243 ts_manager.cc:194] Registered new tserver with Master: 903b03d331eb406da01ced709713f1fe (127.3.221.66:41217)
I20260811 11:53:06.732975 27243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:43547
W20260811 11:53:06.843897 27782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:53:06.844254 27782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:53:06.844300 27782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:53:06.848611 27782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:53:06.848701 27782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:53:06.848764 27782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:53:06.854506 27782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:37931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=45853
--tserver_master_addrs=127.3.221.126:40419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:53:06.855747 27782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:53:06.856810 27782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:53:06.867318 27793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:06.867360 27788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:06.867362 27791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:06.867632 27782 server_base.cc:1034] running on GCE node
I20260811 11:53:06.868125 27782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:53:06.868829 27782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:53:06.870030 27782 hybrid_clock.cc:648] HybridClock initialized: now 1786449186870011 us; error 31 us; skew 500 ppm
Aug 11 11:53:06 dist-test-slave-bndk krb5kdc[26279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449186, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:53:06.872980 27782 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:53:06.873976 27782 webserver.cc:460] Webserver started at http://127.3.221.67:45853/ using document root <none> and password file <none>
I20260811 11:53:06.874565 27782 fs_manager.cc:359] Metadata directory not provided
I20260811 11:53:06.874645 27782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:53:06.878988 27782 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 11:53:06.881207 27806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:06.882629 27782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:53:06.882771 27782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "0efb25eba2184dbfb0e476a285fef172"
format_stamp: "Formatted at 2026-08-11 11:53:05 on dist-test-slave-bndk"
server_key: "05657cc35ad863404415f85d762e2cc1"
server_key_iv: "03e9aaac1fb55650fcd1afbd8330ee25"
server_key_version: "encryptionkey@0"
I20260811 11:53:06.883361 27782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:53:06.901742 27782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:53:06.905414 27782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:53:06.905623 27782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:53:06.906315 27782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:53:06.907328 27782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:53:06.907388 27782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:06.907449 27782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:53:06.907477 27782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:06.917019 27782 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:37931
I20260811 11:53:06.917102 27995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:37931 every 8 connection(s)
I20260811 11:53:06.918289 27782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:53:06.926559 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 27782
Aug 11 11:53:06 dist-test-slave-bndk krb5kdc[26279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449186, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:53:06.934600 27551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:06.920447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:54959 (local address 127.3.221.126:40419)
0811 11:53:06.920628 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:06.920632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:06.921881 (+ 1249us) server_negotiation.cc:407] Connection header received
0811 11:53:06.922816 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:06.922819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:06.922876 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:06.922958 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:06.924314 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.924849 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.926128 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.926550 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.926709 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:06.930272 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:06.930275 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:06.930278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:06.930334 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:06.931811 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:06.932582 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:06.932585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:06.932587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:06.932694 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:06.933175 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:06.933179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:06.933181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:06.933481 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:06.933582 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:06.934269 (+ 687us) server_negotiation.cc:299] Negotiation successful
0811 11:53:06.934428 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260811 11:53:06.935515 28001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:06.920855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.67:54959)
0811 11:53:06.921710 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:06.921757 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:06.922562 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:06.923191 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:06.923199 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:06.923617 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:53:06.924132 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:06.924147 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.924998 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.925001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:06.925973 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:06.925985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.927002 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.927010 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:06.927387 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:06.928263 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:53:06.928291 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:53:06.930068 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:53:06.931991 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:06.932000 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:06.932026 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:06.932430 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:06.932875 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:06.932877 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:06.932879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:06.933056 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:06.933680 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:53:06.933687 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:53:06.933999 (+ 312us) client_negotiation.cc:769] Sending connection context
0811 11:53:06.934187 (+ 188us) client_negotiation.cc:240] Negotiation successful
0811 11:53:06.934578 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":144,"threads_started":1}
I20260811 11:53:06.936684 28002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:06.927502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48522 (local address 127.3.221.126:40419)
0811 11:53:06.927841 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:06.927846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:06.928307 (+ 461us) server_negotiation.cc:407] Connection header received
0811 11:53:06.928439 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:06.928442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:06.928491 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:06.928569 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:06.929463 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.930049 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.931277 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.931525 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.931611 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:06.933402 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:06.933406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:06.933408 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:06.933432 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:06.934941 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:06.935396 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:06.935399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:06.935401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:06.935486 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:06.935776 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:06.935780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:06.935781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:06.935958 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:06.936166 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:06.936362 (+ 196us) server_negotiation.cc:299] Negotiation successful
0811 11:53:06.936527 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"spinlock_wait_cycles":6528,"thread_start_us":162,"threads_started":1}
I20260811 11:53:06.936903 27997 heartbeater.cc:347] Connected to a master server at 127.3.221.126:40419
I20260811 11:53:06.937453 27997 heartbeater.cc:464] Registering TS with master...
I20260811 11:53:06.938092 27997 heartbeater.cc:511] Master 127.3.221.126:40419 requested a full tablet report, sending...
I20260811 11:53:06.939303 27243 ts_manager.cc:194] Registered new tserver with Master: 0efb25eba2184dbfb0e476a285fef172 (127.3.221.67:37931)
I20260811 11:53:06.939881 27243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:54959
I20260811 11:53:06.951825 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:53:06.956847 28002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:06.952599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48530 (local address 127.3.221.126:40419)
0811 11:53:06.952771 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:06.952776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:06.952934 (+ 158us) server_negotiation.cc:407] Connection header received
0811 11:53:06.953120 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:06.953122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:06.953284 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:06.953386 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:53:06.953647 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.954193 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.955081 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.955319 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.955418 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:06.955741 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:53:06.956207 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:53:06.956326 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:06.956606 (+ 280us) server_negotiation.cc:299] Negotiation successful
0811 11:53:06.956682 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 11:53:06.957751 27243 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:53:06.962853 27243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:53:06.974550 27441 tablet_service.cc:1467] Processing CreateTablet for tablet bd5bfe2caad84c1789e0ef6f160920dd (DEFAULT_TABLE table=test-table [id=7cbf6b7bb9044e348598903f2b3b19c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:53:06.975222 27441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5bfe2caad84c1789e0ef6f160920dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:53:06.978164 27375 tablet_bootstrap.cc:492] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24: Bootstrap starting.
I20260811 11:53:06.979157 27375 tablet_bootstrap.cc:654] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24: Neither blocks nor log segments found. Creating new log.
I20260811 11:53:06.979493 27555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:06.973515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:41217 (local address 127.0.0.1:51738)
0811 11:53:06.973637 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:06.973655 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:06.973752 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:06.974926 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:06.974929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:06.974951 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:53:06.975064 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:06.975071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.976226 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.976230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:06.977487 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:06.977501 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.979000 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.979004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:06.979115 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:06.979174 (+ 59us) client_negotiation.cc:769] Sending connection context
0811 11:53:06.979233 (+ 59us) client_negotiation.cc:240] Negotiation successful
0811 11:53:06.979336 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260811 11:53:06.980449 28011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:06.973694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51738 (local address 127.3.221.66:41217)
0811 11:53:06.974406 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:06.974419 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:06.974446 (+ 27us) server_negotiation.cc:407] Connection header received
0811 11:53:06.974533 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:06.974538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:06.974731 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:06.974908 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:53:06.975207 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.976080 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.977742 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.978840 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.979855 (+ 1015us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:06.980073 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:06.980156 (+ 83us) server_negotiation.cc:299] Negotiation successful
0811 11:53:06.980285 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":143,"threads_started":1}
I20260811 11:53:06.980921 27375 tablet_bootstrap.cc:492] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24: No bootstrap required, opened a new log
I20260811 11:53:06.981027 27375 ts_tablet_manager.cc:1397] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260811 11:53:06.981976 27375 raft_consensus.cc:348] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:06.982095 27375 raft_consensus.cc:374] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:53:06.982126 27375 raft_consensus.cc:729] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc6b1d1884454450ae1f9cbc66519a24, State: Initialized, Role: FOLLOWER
I20260811 11:53:06.982297 27375 consensus_queue.cc:260] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:06.982592 27375 ts_tablet_manager.cc:1428] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260811 11:53:06.984076 27673 tablet_service.cc:1467] Processing CreateTablet for tablet bd5bfe2caad84c1789e0ef6f160920dd (DEFAULT_TABLE table=test-table [id=7cbf6b7bb9044e348598903f2b3b19c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:53:06.985426 28014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:06.973984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:37931 (local address 127.0.0.1:38682)
0811 11:53:06.974255 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:06.974270 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:06.974365 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:06.980058 (+ 5693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:06.980062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:06.980084 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:53:06.980161 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:06.980166 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.981602 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.981606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:06.983664 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:06.983680 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.984866 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.984870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:06.984986 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:06.985013 (+ 27us) client_negotiation.cc:769] Sending connection context
0811 11:53:06.985102 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 11:53:06.985198 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":109,"threads_started":1}
I20260811 11:53:06.985579 28016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:06.974158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38682 (local address 127.3.221.67:37931)
0811 11:53:06.979042 (+ 4884us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:06.979055 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:06.979084 (+ 29us) server_negotiation.cc:407] Connection header received
0811 11:53:06.979179 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:06.979185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:06.979677 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:06.979862 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:53:06.980285 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.981438 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.983859 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:06.984528 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:06.984850 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:06.985029 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:06.985138 (+ 109us) server_negotiation.cc:299] Negotiation successful
0811 11:53:06.985276 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4749,"thread_start_us":154,"threads_started":1}
I20260811 11:53:06.985935 27673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5bfe2caad84c1789e0ef6f160920dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:53:06.988703 27882 tablet_service.cc:1467] Processing CreateTablet for tablet bd5bfe2caad84c1789e0ef6f160920dd (DEFAULT_TABLE table=test-table [id=7cbf6b7bb9044e348598903f2b3b19c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:53:06.989825 27882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5bfe2caad84c1789e0ef6f160920dd. 1 dirs total, 0 dirs full, 0 dirs failed
W20260811 11:53:06.990502 27548 tablet.cc:2369] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:53:06.994235 28019 tablet_bootstrap.cc:492] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe: Bootstrap starting.
I20260811 11:53:06.996793 28019 tablet_bootstrap.cc:654] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe: Neither blocks nor log segments found. Creating new log.
I20260811 11:53:06.997737 28019 log.cc:824] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe: Log is configured to *not* fsync() on all Append() calls
I20260811 11:53:06.998148 28020 tablet_bootstrap.cc:492] T bd5bfe2caad84c1789e0ef6f160920dd P 0efb25eba2184dbfb0e476a285fef172: Bootstrap starting.
I20260811 11:53:07.000490 28019 tablet_bootstrap.cc:492] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe: No bootstrap required, opened a new log
I20260811 11:53:07.000731 28019 ts_tablet_manager.cc:1397] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260811 11:53:07.001319 28020 tablet_bootstrap.cc:654] T bd5bfe2caad84c1789e0ef6f160920dd P 0efb25eba2184dbfb0e476a285fef172: Neither blocks nor log segments found. Creating new log.
I20260811 11:53:07.002245 28020 log.cc:824] T bd5bfe2caad84c1789e0ef6f160920dd P 0efb25eba2184dbfb0e476a285fef172: Log is configured to *not* fsync() on all Append() calls
I20260811 11:53:07.004715 28020 tablet_bootstrap.cc:492] T bd5bfe2caad84c1789e0ef6f160920dd P 0efb25eba2184dbfb0e476a285fef172: No bootstrap required, opened a new log
I20260811 11:53:07.004962 28020 ts_tablet_manager.cc:1397] T bd5bfe2caad84c1789e0ef6f160920dd P 0efb25eba2184dbfb0e476a285fef172: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260811 11:53:07.004899 28019 raft_consensus.cc:348] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:07.005270 28019 raft_consensus.cc:374] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:53:07.005347 28019 raft_consensus.cc:729] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903b03d331eb406da01ced709713f1fe, State: Initialized, Role: FOLLOWER
I20260811 11:53:07.005849 28019 consensus_queue.cc:260] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:07.006592 27770 heartbeater.cc:503] Master 127.3.221.126:40419 was elected leader, sending a full tablet report...
I20260811 11:53:07.007021 28019 ts_tablet_manager.cc:1428] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260811 11:53:07.008744 28020 raft_consensus.cc:348] T bd5bfe2caad84c1789e0ef6f160920dd P 0efb25eba2184dbfb0e476a285fef172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:07.009013 28020 raft_consensus.cc:374] T bd5bfe2caad84c1789e0ef6f160920dd P 0efb25eba2184dbfb0e476a285fef172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:53:07.009073 28020 raft_consensus.cc:729] T bd5bfe2caad84c1789e0ef6f160920dd P 0efb25eba2184dbfb0e476a285fef172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0efb25eba2184dbfb0e476a285fef172, State: Initialized, Role: FOLLOWER
I20260811 11:53:07.009560 28020 consensus_queue.cc:260] T bd5bfe2caad84c1789e0ef6f160920dd P 0efb25eba2184dbfb0e476a285fef172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:07.010241 27997 heartbeater.cc:503] Master 127.3.221.126:40419 was elected leader, sending a full tablet report...
I20260811 11:53:07.010645 28020 ts_tablet_manager.cc:1428] T bd5bfe2caad84c1789e0ef6f160920dd P 0efb25eba2184dbfb0e476a285fef172: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260811 11:53:07.169530 27998 tablet.cc:2369] T bd5bfe2caad84c1789e0ef6f160920dd P 0efb25eba2184dbfb0e476a285fef172: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:53:07.212865 27774 tablet.cc:2369] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:53:07.377974 28028 raft_consensus.cc:482] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:53:07.378224 28028 raft_consensus.cc:504] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:07.379662 28028 leader_election.cc:288] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0efb25eba2184dbfb0e476a285fef172 (127.3.221.67:37931), bc6b1d1884454450ae1f9cbc66519a24 (127.3.221.65:32839)
I20260811 11:53:07.384537 28031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:07.379803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:37931 (local address 127.3.221.66:40633)
0811 11:53:07.380497 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:07.380518 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:07.380864 (+ 346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:07.381251 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:07.381254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:07.381273 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:53:07.381433 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:07.381441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:07.382263 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:07.382267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:07.383243 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:07.383255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:07.384045 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:07.384049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:07.384151 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:07.384182 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 11:53:07.384258 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 11:53:07.384349 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":141,"threads_started":1}
I20260811 11:53:07.384542 28016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:07.380132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:40633 (local address 127.3.221.67:37931)
0811 11:53:07.380361 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:07.380367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:07.380678 (+ 311us) server_negotiation.cc:407] Connection header received
0811 11:53:07.381060 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:07.381064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:07.381114 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:07.381178 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:53:07.381561 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:07.382117 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:07.383430 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:07.383903 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:07.384008 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:07.384056 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:07.384321 (+ 265us) server_negotiation.cc:299] Negotiation successful
0811 11:53:07.384408 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260811 11:53:07.384742 28030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:07.380146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:32839 (local address 127.3.221.66:41055)
0811 11:53:07.380730 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:07.380757 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:07.380917 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:07.381416 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:07.381419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:07.381440 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:53:07.381578 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:07.381586 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:07.382606 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:07.382610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:07.383490 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:07.383501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:07.384339 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:07.384343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:07.384443 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:07.384467 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 11:53:07.384541 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 11:53:07.384613 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":86,"threads_started":1}
I20260811 11:53:07.385166 28032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:07.380351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:41055 (local address 127.3.221.65:32839)
0811 11:53:07.380681 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:07.380687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:07.380945 (+ 258us) server_negotiation.cc:407] Connection header received
0811 11:53:07.381042 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:07.381045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:07.381117 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:07.381267 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:53:07.381731 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:07.382465 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:07.383634 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:07.384160 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:07.384295 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:07.384475 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:07.384558 (+ 83us) server_negotiation.cc:299] Negotiation successful
0811 11:53:07.385028 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":87,"threads_started":1}
I20260811 11:53:07.385278 27916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5bfe2caad84c1789e0ef6f160920dd" candidate_uuid: "903b03d331eb406da01ced709713f1fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0efb25eba2184dbfb0e476a285fef172" is_pre_election: true
I20260811 11:53:07.385651 27916 raft_consensus.cc:2454] T bd5bfe2caad84c1789e0ef6f160920dd P 0efb25eba2184dbfb0e476a285fef172 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 903b03d331eb406da01ced709713f1fe in term 0.
I20260811 11:53:07.385713 27476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5bfe2caad84c1789e0ef6f160920dd" candidate_uuid: "903b03d331eb406da01ced709713f1fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc6b1d1884454450ae1f9cbc66519a24" is_pre_election: true
I20260811 11:53:07.386027 27476 raft_consensus.cc:2454] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 903b03d331eb406da01ced709713f1fe in term 0.
I20260811 11:53:07.386179 27589 leader_election.cc:302] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0efb25eba2184dbfb0e476a285fef172, 903b03d331eb406da01ced709713f1fe; no voters:
I20260811 11:53:07.386557 28028 raft_consensus.cc:2781] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:53:07.386668 28028 raft_consensus.cc:482] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:53:07.386723 28028 raft_consensus.cc:3037] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:53:07.387972 28028 raft_consensus.cc:504] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:07.388365 28028 leader_election.cc:288] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe [CANDIDATE]: Term 1 election: Requested vote from peers 0efb25eba2184dbfb0e476a285fef172 (127.3.221.67:37931), bc6b1d1884454450ae1f9cbc66519a24 (127.3.221.65:32839)
I20260811 11:53:07.388785 27476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5bfe2caad84c1789e0ef6f160920dd" candidate_uuid: "903b03d331eb406da01ced709713f1fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc6b1d1884454450ae1f9cbc66519a24"
I20260811 11:53:07.388811 27916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5bfe2caad84c1789e0ef6f160920dd" candidate_uuid: "903b03d331eb406da01ced709713f1fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0efb25eba2184dbfb0e476a285fef172"
I20260811 11:53:07.388922 27916 raft_consensus.cc:3037] T bd5bfe2caad84c1789e0ef6f160920dd P 0efb25eba2184dbfb0e476a285fef172 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:53:07.388933 27476 raft_consensus.cc:3037] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:53:07.390064 27476 raft_consensus.cc:2454] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 903b03d331eb406da01ced709713f1fe in term 1.
I20260811 11:53:07.390069 27916 raft_consensus.cc:2454] T bd5bfe2caad84c1789e0ef6f160920dd P 0efb25eba2184dbfb0e476a285fef172 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 903b03d331eb406da01ced709713f1fe in term 1.
I20260811 11:53:07.390398 27589 leader_election.cc:302] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0efb25eba2184dbfb0e476a285fef172, 903b03d331eb406da01ced709713f1fe, bc6b1d1884454450ae1f9cbc66519a24; no voters:
I20260811 11:53:07.390574 28028 raft_consensus.cc:2781] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:53:07.390795 28028 raft_consensus.cc:686] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe [term 1 LEADER]: Becoming Leader. State: Replica: 903b03d331eb406da01ced709713f1fe, State: Running, Role: LEADER
I20260811 11:53:07.391155 28028 consensus_queue.cc:237] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:07.395521 27243 catalog_manager.cc:5495] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 903b03d331eb406da01ced709713f1fe (127.3.221.66). New cstate: current_term: 1 leader_uuid: "903b03d331eb406da01ced709713f1fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } health_report { overall_health: UNKNOWN } } }
I20260811 11:53:07.443933 28011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:07.440186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51742 (local address 127.3.221.66:41217)
0811 11:53:07.440302 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:07.440305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:07.440324 (+ 19us) server_negotiation.cc:407] Connection header received
0811 11:53:07.440411 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:07.440414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:07.440473 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:07.440572 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:53:07.440858 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:07.441442 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:07.442270 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:07.442497 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:07.442595 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:07.442973 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:53:07.443446 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:53:07.443616 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:07.443657 (+ 41us) server_negotiation.cc:299] Negotiation successful
0811 11:53:07.443797 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260811 11:53:07.452172 27476 raft_consensus.cc:1261] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Refusing update from remote peer 903b03d331eb406da01ced709713f1fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:53:07.452178 27916 raft_consensus.cc:1261] T bd5bfe2caad84c1789e0ef6f160920dd P 0efb25eba2184dbfb0e476a285fef172 [term 1 FOLLOWER]: Refusing update from remote peer 903b03d331eb406da01ced709713f1fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:53:07.453045 28033 consensus_queue.cc:1035] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:53:07.453311 28028 consensus_queue.cc:1035] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:53:07.459510 28036 mvcc.cc:205] Tried to move back new op lower bound from 7317295871796117504 to 7317295871556915200. Current Snapshot: MvccSnapshot[applied={T|T < 7317295871796117504}]
I20260811 11:53:07.461460 28037 mvcc.cc:205] Tried to move back new op lower bound from 7317295871796117504 to 7317295871556915200. Current Snapshot: MvccSnapshot[applied={T|T < 7317295871796117504 or (T in {7317295871796117504})}]
I20260811 11:53:07.469619 27243 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:53:07.469820 27243 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:53:07.472477 27243 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82: Sending DeleteTablet for 3 replicas of tablet bd5bfe2caad84c1789e0ef6f160920dd
I20260811 11:53:07.473291 27441 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5bfe2caad84c1789e0ef6f160920dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:53:07 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:34878
I20260811 11:53:07.473323 27673 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5bfe2caad84c1789e0ef6f160920dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:53:07 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:51738
I20260811 11:53:07.473356 27882 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5bfe2caad84c1789e0ef6f160920dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:53:07 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:38682
I20260811 11:53:07.473836 28051 tablet_replica.cc:330] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24: stopping tablet replica
I20260811 11:53:07.473971 28051 raft_consensus.cc:2229] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:53:07.474092 28051 raft_consensus.cc:2258] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:53:07.474107 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 27307
I20260811 11:53:07.504830 28060 tablet_replica.cc:330] T bd5bfe2caad84c1789e0ef6f160920dd P 0efb25eba2184dbfb0e476a285fef172: stopping tablet replica
I20260811 11:53:07.505431 28060 raft_consensus.cc:2229] T bd5bfe2caad84c1789e0ef6f160920dd P 0efb25eba2184dbfb0e476a285fef172 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:53:07.506014 28060 raft_consensus.cc:2258] T bd5bfe2caad84c1789e0ef6f160920dd P 0efb25eba2184dbfb0e476a285fef172 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:53:07.506510 28055 tablet_replica.cc:330] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe: stopping tablet replica
I20260811 11:53:07.506837 28055 raft_consensus.cc:2229] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:53:07.507282 28055 raft_consensus.cc:2258] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:53:07.508289 28060 ts_tablet_manager.cc:1905] T bd5bfe2caad84c1789e0ef6f160920dd P 0efb25eba2184dbfb0e476a285fef172: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:53:07.508661 28055 ts_tablet_manager.cc:1905] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:53:07.511641 28060 ts_tablet_manager.cc:1918] T bd5bfe2caad84c1789e0ef6f160920dd P 0efb25eba2184dbfb0e476a285fef172: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:53:07.511862 28060 log.cc:1196] T bd5bfe2caad84c1789e0ef6f160920dd P 0efb25eba2184dbfb0e476a285fef172: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/bd5bfe2caad84c1789e0ef6f160920dd
I20260811 11:53:07.511888 28055 ts_tablet_manager.cc:1918] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:53:07.512017 28055 log.cc:1196] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/bd5bfe2caad84c1789e0ef6f160920dd
I20260811 11:53:07.512374 28060 ts_tablet_manager.cc:1939] T bd5bfe2caad84c1789e0ef6f160920dd P 0efb25eba2184dbfb0e476a285fef172: Deleting consensus metadata
I20260811 11:53:07.512401 28055 ts_tablet_manager.cc:1939] T bd5bfe2caad84c1789e0ef6f160920dd P 903b03d331eb406da01ced709713f1fe: Deleting consensus metadata
W20260811 11:53:07.513594 27218 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.221.65:32839 (error 108)
I20260811 11:53:07.513932 27222 catalog_manager.cc:4841] TS 0efb25eba2184dbfb0e476a285fef172 (127.3.221.67:37931): tablet bd5bfe2caad84c1789e0ef6f160920dd (table test-table [id=7cbf6b7bb9044e348598903f2b3b19c3]) successfully deleted
I20260811 11:53:07.514127 27221 catalog_manager.cc:4841] TS 903b03d331eb406da01ced709713f1fe (127.3.221.66:41217): tablet bd5bfe2caad84c1789e0ef6f160920dd (table test-table [id=7cbf6b7bb9044e348598903f2b3b19c3]) successfully deleted
W20260811 11:53:07.514930 28063 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:53:07.514477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:32839 (local address 127.0.0.1:47220)
0811 11:53:07.514749 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:07.514821 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.65:32839: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":61,"threads_started":1}
I20260811 11:53:07.515002 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 27553
W20260811 11:53:07.515050 27218 catalog_manager.cc:4584] TS bc6b1d1884454450ae1f9cbc66519a24 (127.3.221.65:32839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd5bfe2caad84c1789e0ef6f160920dd: Network error: Client connection negotiation failed: client connection to 127.3.221.65:32839: connect: Connection reset by peer (error 104)
I20260811 11:53:07.527989 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 27782
I20260811 11:53:07.546720 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 27187
I20260811 11:53:07.562822 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:40419
--webserver_interface=127.3.221.126
--webserver_port=43831
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:40419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:53:07.691558 28065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:53:07.691854 28065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:53:07.691903 28065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:53:07.696965 28065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:53:07.697062 28065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:53:07.697090 28065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:53:07.697114 28065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:53:07.697144 28065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:53:07.705122 28065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:40419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:40419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=43831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:53:07.706418 28065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:53:07.707535 28065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:53:07.716948 28092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:07.716953 28088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:07.716953 28087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:07.717207 28065 server_base.cc:1034] running on GCE node
I20260811 11:53:07.718111 28065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:53:07.719364 28065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:53:07.720662 28065 hybrid_clock.cc:648] HybridClock initialized: now 1786449187720642 us; error 37 us; skew 500 ppm
Aug 11 11:53:07 dist-test-slave-bndk krb5kdc[26279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449187, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:53:07.724059 28065 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:53:07.725149 28065 webserver.cc:460] Webserver started at http://127.3.221.126:43831/ using document root <none> and password file <none>
I20260811 11:53:07.725864 28065 fs_manager.cc:359] Metadata directory not provided
I20260811 11:53:07.726105 28065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:53:07.731076 28065 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 11:53:07.733809 28103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:07.735402 28065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:53:07.735558 28065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "691ca47614cf4870a72b077bdc797e82"
format_stamp: "Formatted at 2026-08-11 11:53:04 on dist-test-slave-bndk"
server_key: "830487391d2e4b55a211fd099941d060"
server_key_iv: "c8ef99263eaba4e050d7ad6383ad42ee"
server_key_version: "encryptionkey@0"
I20260811 11:53:07.736308 28065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:53:07.749019 28065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:53:07.752419 28065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:53:07.752621 28065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:53:07.760309 28065 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:40419
I20260811 11:53:07.760655 28183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:40419 every 8 connection(s)
I20260811 11:53:07.761574 28065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:53:07.761932 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 28065
I20260811 11:53:07.762737 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:32839
--local_ip_for_outbound_sockets=127.3.221.65
--tserver_master_addrs=127.3.221.126:40419
--webserver_port=42685
--webserver_interface=127.3.221.65
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:53:07.767429 28185 sys_catalog.cc:263] Verifying existing consensus state
I20260811 11:53:07.769657 28185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82: Bootstrap starting.
I20260811 11:53:07.790728 28185 log.cc:824] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82: Log is configured to *not* fsync() on all Append() calls
I20260811 11:53:07.804633 28185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:53:07.805080 28185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82: Bootstrap complete.
I20260811 11:53:07.809970 28185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691ca47614cf4870a72b077bdc797e82" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } }
I20260811 11:53:07.810421 28185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691ca47614cf4870a72b077bdc797e82, State: Initialized, Role: FOLLOWER
I20260811 11:53:07.810945 28185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691ca47614cf4870a72b077bdc797e82" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } }
I20260811 11:53:07.811072 28185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:53:07.811141 28185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:53:07.811259 28185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 2 FOLLOWER]: Advancing to term 3
I20260811 11:53:07.813372 28185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691ca47614cf4870a72b077bdc797e82" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } }
I20260811 11:53:07.813720 28185 leader_election.cc:302] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 691ca47614cf4870a72b077bdc797e82; no voters:
I20260811 11:53:07.814054 28185 leader_election.cc:288] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260811 11:53:07.814203 28191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 3 FOLLOWER]: Leader election won for term 3
I20260811 11:53:07.814463 28191 raft_consensus.cc:686] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [term 3 LEADER]: Becoming Leader. State: Replica: 691ca47614cf4870a72b077bdc797e82, State: Running, Role: LEADER
I20260811 11:53:07.814831 28191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691ca47614cf4870a72b077bdc797e82" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } }
I20260811 11:53:07.815378 28185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:53:07.815984 28193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 691ca47614cf4870a72b077bdc797e82. Latest consensus state: current_term: 3 leader_uuid: "691ca47614cf4870a72b077bdc797e82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691ca47614cf4870a72b077bdc797e82" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } } }
I20260811 11:53:07.816129 28193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [sys.catalog]: This master's current role is: LEADER
I20260811 11:53:07.816774 28192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "691ca47614cf4870a72b077bdc797e82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691ca47614cf4870a72b077bdc797e82" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } } }
I20260811 11:53:07.816877 28192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691ca47614cf4870a72b077bdc797e82 [sys.catalog]: This master's current role is: LEADER
I20260811 11:53:07.817497 28197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:53:07.823101 28197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:53:07.823814 28197 catalog_manager.cc:1259] Loaded cluster ID: 94cac514bcb94264a4c93de5648b35ba
I20260811 11:53:07.823884 28197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260811 11:53:07.825032 28197 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbceaf2c9ad google::LogMessage::Fail() at ??:0
@ 0x7fbceaf2f88a google::LogMessage::SendToLog() at ??:0
@ 0x7fbceaf2c4de google::LogMessage::Flush() at ??:0
@ 0x7fbceaf2da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbcf586bfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fbcf586cd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbcf5869579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fbcf58ad640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbced17e3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbced17ec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fbced1803f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbced16f512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbcee7b36db start_thread at ??:0
@ 0x7fbce9a2a61f clone at ??:0
W20260811 11:53:07.917515 28188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:53:07.917866 28188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:53:07.917935 28188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:53:07.922540 28188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:53:07.922616 28188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:53:07.922703 28188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:53:07.929572 28188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:32839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=42685
--tserver_master_addrs=127.3.221.126:40419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:53:07.930815 28188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:53:07.931954 28188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:53:07.943071 28249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:07.943071 28248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:07.943131 28188 server_base.cc:1034] running on GCE node
W20260811 11:53:07.943459 28251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:07.943934 28188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:53:07.944756 28188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:53:07.945948 28188 hybrid_clock.cc:648] HybridClock initialized: now 1786449187945928 us; error 40 us; skew 500 ppm
Aug 11 11:53:07 dist-test-slave-bndk krb5kdc[26279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449187, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:53:07.949066 28188 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:53:07.950243 28188 webserver.cc:460] Webserver started at http://127.3.221.65:42685/ using document root <none> and password file <none>
I20260811 11:53:07.950840 28188 fs_manager.cc:359] Metadata directory not provided
I20260811 11:53:07.950917 28188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:53:07.955065 28188 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260811 11:53:07.957484 28268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:07.959120 28188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 11:53:07.959260 28188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "bc6b1d1884454450ae1f9cbc66519a24"
format_stamp: "Formatted at 2026-08-11 11:53:05 on dist-test-slave-bndk"
server_key: "c92c5d7a16739b8ed3323f88351d5264"
server_key_iv: "310a2536254248c76216b2c8eb1e99ed"
server_key_version: "encryptionkey@0"
I20260811 11:53:07.959874 28188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:53:07.983057 28188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:53:07.987813 28188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:53:07.988085 28188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:53:07.988945 28188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:53:07.990845 28286 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260811 11:53:07.995605 28188 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 11:53:07.995695 28188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260811 11:53:07.995759 28188 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 11:53:07.997679 28188 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 11:53:07.997752 28188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260811 11:53:07.998061 28286 tablet_bootstrap.cc:492] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24: Bootstrap starting.
I20260811 11:53:08.009282 28188 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:32839
I20260811 11:53:08.009631 28482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:32839 every 8 connection(s)
I20260811 11:53:08.010857 28188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:53:08.018694 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 28188
I20260811 11:53:08.019212 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:41217
--local_ip_for_outbound_sockets=127.3.221.66
--tserver_master_addrs=127.3.221.126:40419
--webserver_port=34647
--webserver_interface=127.3.221.66
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:53:08.020164 28286 log.cc:824] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24: Log is configured to *not* fsync() on all Append() calls
I20260811 11:53:08.030551 28286 tablet_bootstrap.cc:492] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:53:08.031044 28286 tablet_bootstrap.cc:492] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24: Bootstrap complete.
I20260811 11:53:08.032516 28286 ts_tablet_manager.cc:1397] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24: Time spent bootstrapping tablet: real 0.035s user 0.026s sys 0.005s
I20260811 11:53:08.036242 28286 raft_consensus.cc:348] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:08.036877 28286 raft_consensus.cc:729] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc6b1d1884454450ae1f9cbc66519a24, State: Initialized, Role: FOLLOWER
I20260811 11:53:08.037839 28286 consensus_queue.cc:260] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:08.039077 28286 ts_tablet_manager.cc:1428] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24: Time spent starting tablet: real 0.006s user 0.009s sys 0.000s
W20260811 11:53:08.114392 28485 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:53:08.013923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.65:59025)
0811 11:53:08.014575 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:08.014642 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:08.015784 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:08.113170 (+ 97386us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":170,"threads_started":1}
W20260811 11:53:08.115192 28483 heartbeater.cc:644] Failed to heartbeat to 127.3.221.126:40419 (0 consecutive failures): Network error: Failed to ping master at 127.3.221.126:40419: Client connection negotiation failed: client connection to 127.3.221.126:40419: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260811 11:53:08.116122 28485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:08.115780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.65:49139)
0811 11:53:08.115940 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:08.116036 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260811 11:53:08.117074 28485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:08.116782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.65:39357)
0811 11:53:08.116938 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:08.116999 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260811 11:53:08.117344 28483 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 11:53:08.156674 28489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:53:08.156980 28489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:53:08.157043 28489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:53:08.161276 28489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:53:08.161352 28489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:53:08.161439 28489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:53:08.167546 28489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:41217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=34647
--tserver_master_addrs=127.3.221.126:40419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:53:08.168936 28489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:53:08.170300 28489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:53:08.180934 28499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:08.180934 28498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:08.180958 28505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:08.181248 28489 server_base.cc:1034] running on GCE node
I20260811 11:53:08.181701 28489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:53:08.182548 28489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:53:08.183756 28489 hybrid_clock.cc:648] HybridClock initialized: now 1786449188183740 us; error 29 us; skew 500 ppm
Aug 11 11:53:08 dist-test-slave-bndk krb5kdc[26279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449188, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:53:08.186985 28489 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:53:08.188014 28489 webserver.cc:460] Webserver started at http://127.3.221.66:34647/ using document root <none> and password file <none>
I20260811 11:53:08.188596 28489 fs_manager.cc:359] Metadata directory not provided
I20260811 11:53:08.188678 28489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:53:08.192945 28489 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260811 11:53:08.195447 28512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:08.196893 28489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 11:53:08.197012 28489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "903b03d331eb406da01ced709713f1fe"
format_stamp: "Formatted at 2026-08-11 11:53:05 on dist-test-slave-bndk"
server_key: "65bd0c4a5c31add6f734b499f31f9aba"
server_key_iv: "ddcab6a90e1c315d970cc78d183a9f6a"
server_key_version: "encryptionkey@0"
I20260811 11:53:08.197692 28489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:53:08.223664 28489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:53:08.227015 28489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:53:08.227192 28489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:53:08.227813 28489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:53:08.228688 28489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:53:08.228741 28489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:08.228782 28489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:53:08.228796 28489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:08.238461 28489 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:41217
I20260811 11:53:08.238551 28708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:41217 every 8 connection(s)
I20260811 11:53:08.239602 28489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:53:08.244210 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:08.242079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.66:59921)
0811 11:53:08.243245 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:08.243436 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":945,"thread_start_us":361,"threads_started":1}
W20260811 11:53:08.244895 28709 heartbeater.cc:644] Failed to heartbeat to 127.3.221.126:40419 (0 consecutive failures): Network error: Failed to ping master at 127.3.221.126:40419: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
I20260811 11:53:08.245823 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:08.245515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.66:60053)
0811 11:53:08.245694 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:08.245754 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260811 11:53:08.246915 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:08.246660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.66:40763)
0811 11:53:08.246803 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:08.246854 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260811 11:53:08.247493 28709 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 11:53:08.247864 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 28489
I20260811 11:53:08.248377 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:37931
--local_ip_for_outbound_sockets=127.3.221.67
--tserver_master_addrs=127.3.221.126:40419
--webserver_port=45853
--webserver_interface=127.3.221.67
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:53:08.375722 28713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:53:08.376091 28713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:53:08.376142 28713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:53:08.380647 28713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:53:08.380740 28713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:53:08.380805 28713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:53:08.386536 28713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:37931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=45853
--tserver_master_addrs=127.3.221.126:40419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:53:08.387794 28713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:53:08.388914 28713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:53:08.399295 28719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:08.399328 28725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:08.399354 28718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:08.399715 28713 server_base.cc:1034] running on GCE node
I20260811 11:53:08.400156 28713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:53:08.400899 28713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:53:08.402076 28713 hybrid_clock.cc:648] HybridClock initialized: now 1786449188402032 us; error 60 us; skew 500 ppm
Aug 11 11:53:08 dist-test-slave-bndk krb5kdc[26279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449188, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:53:08.408406 28713 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:53:08.409492 28713 webserver.cc:460] Webserver started at http://127.3.221.67:45853/ using document root <none> and password file <none>
I20260811 11:53:08.410066 28713 fs_manager.cc:359] Metadata directory not provided
I20260811 11:53:08.410130 28713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:53:08.414469 28713 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260811 11:53:08.417016 28733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:08.418602 28713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 11:53:08.418728 28713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "0efb25eba2184dbfb0e476a285fef172"
format_stamp: "Formatted at 2026-08-11 11:53:05 on dist-test-slave-bndk"
server_key: "05657cc35ad863404415f85d762e2cc1"
server_key_iv: "03e9aaac1fb55650fcd1afbd8330ee25"
server_key_version: "encryptionkey@0"
I20260811 11:53:08.419337 28713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:53:08.439422 28713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:53:08.443138 28713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:53:08.443523 28713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:53:08.444878 28713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:53:08.448145 28713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:53:08.448222 28713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:08.448274 28713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:53:08.448295 28713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:08.458081 28713 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:37931
I20260811 11:53:08.458149 28932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:37931 every 8 connection(s)
I20260811 11:53:08.459151 28713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:53:08.463608 28938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:08.461581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.67:41175)
0811 11:53:08.462579 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:08.462732 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":152,"threads_started":1}
W20260811 11:53:08.464298 28935 heartbeater.cc:644] Failed to heartbeat to 127.3.221.126:40419 (0 consecutive failures): Network error: Failed to ping master at 127.3.221.126:40419: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
I20260811 11:53:08.465005 28938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:08.464769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.67:60725)
0811 11:53:08.464891 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:08.464948 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260811 11:53:08.466122 28938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:08.465876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.67:34855)
0811 11:53:08.466017 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:08.466068 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260811 11:53:08.466320 28935 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 11:53:08.468241 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 28713
I20260811 11:53:09.118629 28940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:09.118070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.65:32973)
0811 11:53:09.118425 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:09.118504 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":102,"threads_started":1}
I20260811 11:53:09.248771 28942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:09.248171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.66:57225)
0811 11:53:09.248576 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:09.248648 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":130,"threads_started":1}
I20260811 11:53:09.467636 28946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:09.467073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.67:49893)
0811 11:53:09.467430 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:09.467505 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":81,"threads_started":1}
I20260811 11:53:09.852916 28947 raft_consensus.cc:482] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:53:09.853150 28947 raft_consensus.cc:504] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:09.854411 28947 leader_election.cc:288] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 903b03d331eb406da01ced709713f1fe (127.3.221.66:41217), 0efb25eba2184dbfb0e476a285fef172 (127.3.221.67:37931)
Aug 11 11:53:09 dist-test-slave-bndk krb5kdc[26279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449187, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.67@KRBTEST.COM
Aug 11 11:53:09 dist-test-slave-bndk krb5kdc[26279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449187, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.66@KRBTEST.COM
I20260811 11:53:09.869831 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:09.854853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:38803 (local address 127.3.221.67:37931)
0811 11:53:09.855278 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:09.855301 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:09.855337 (+ 36us) server_negotiation.cc:407] Connection header received
0811 11:53:09.856005 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:09.856010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:09.856368 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:09.856656 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:09.858067 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:09.859065 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:09.860309 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:09.860805 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:09.860966 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:09.864000 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:09.864008 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:09.864026 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:09.864064 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:09.865921 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:09.866876 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:09.866882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:09.866888 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:09.867049 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:09.867481 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:09.867485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:09.867487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:09.868039 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:09.868149 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:09.869305 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0811 11:53:09.869631 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":66,"threads_started":1}
I20260811 11:53:09.869815 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:09.854633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:41217 (local address 127.3.221.65:48795)
0811 11:53:09.854957 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:09.854972 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:09.855094 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:09.856743 (+ 1649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:09.856749 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:09.857357 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:53:09.857886 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:09.857905 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:09.859216 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:09.859219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:09.860174 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:09.860186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:09.860942 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:09.860946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:09.861193 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:09.861903 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:53:09.861920 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:53:09.863926 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:53:09.867208 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:09.867211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:09.867213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:09.867400 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:09.867743 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:09.867746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:09.867747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:09.867836 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:09.869029 (+ 1193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:53:09.869032 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:53:09.869179 (+ 147us) client_negotiation.cc:769] Sending connection context
0811 11:53:09.869326 (+ 147us) client_negotiation.cc:240] Negotiation successful
0811 11:53:09.869620 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":142,"spinlock_wait_cycles":3072,"thread_start_us":98,"threads_started":1}
I20260811 11:53:09.869957 28949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:09.854662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:37931 (local address 127.3.221.65:38803)
0811 11:53:09.855103 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:09.855118 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:09.855215 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:09.856752 (+ 1537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:09.856755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:09.857357 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:53:09.857886 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:09.857904 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:09.859220 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:09.859223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:09.860177 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:09.860188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:09.860937 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:09.860941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:09.861167 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:09.861890 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:53:09.861908 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:53:09.863669 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:53:09.866146 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:09.866160 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:09.866189 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:09.866690 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:09.867163 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:09.867167 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:09.867169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:09.867356 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:09.868297 (+ 941us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:53:09.868324 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:53:09.868811 (+ 487us) client_negotiation.cc:769] Sending connection context
0811 11:53:09.869161 (+ 350us) client_negotiation.cc:240] Negotiation successful
0811 11:53:09.869732 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":283,"spinlock_wait_cycles":27904,"thread_start_us":61,"threads_started":1}
I20260811 11:53:09.869987 28951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:09.854798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:48795 (local address 127.3.221.66:41217)
0811 11:53:09.855263 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:09.855276 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:09.855316 (+ 40us) server_negotiation.cc:407] Connection header received
0811 11:53:09.856004 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:09.856010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:09.856368 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:09.856656 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:09.858066 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:09.859065 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:09.860307 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:09.860805 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:09.860967 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:09.864116 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:09.864122 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:09.864134 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:09.864166 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:09.867100 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:09.867521 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:09.867525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:09.867530 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:09.867648 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:09.867921 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:09.867923 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:09.867924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:09.868827 (+ 903us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:09.869137 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:09.869520 (+ 383us) server_negotiation.cc:299] Negotiation successful
0811 11:53:09.869815 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":100,"threads_started":1}
W20260811 11:53:09.872835 28275 leader_election.cc:341] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0efb25eba2184dbfb0e476a285fef172 (127.3.221.67:37931): Not found: Tablet not found: bd5bfe2caad84c1789e0ef6f160920dd
W20260811 11:53:09.873061 28272 leader_election.cc:341] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 903b03d331eb406da01ced709713f1fe (127.3.221.66:41217): Not found: Tablet not found: bd5bfe2caad84c1789e0ef6f160920dd
I20260811 11:53:09.873178 28272 leader_election.cc:302] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc6b1d1884454450ae1f9cbc66519a24; no voters: 0efb25eba2184dbfb0e476a285fef172, 903b03d331eb406da01ced709713f1fe
I20260811 11:53:09.873494 28947 raft_consensus.cc:2726] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:53:10.119952 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:10.119614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.65:38249)
0811 11:53:10.119805 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:10.119864 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260811 11:53:10.250295 28954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:10.249776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.66:59545)
0811 11:53:10.250096 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:10.250166 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":92,"threads_started":1}
I20260811 11:53:10.469009 28955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:10.468528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.67:60421)
0811 11:53:10.468841 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:10.468908 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":79,"threads_started":1}
I20260811 11:53:11.121374 28956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:11.120841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.65:51059)
0811 11:53:11.121143 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:11.121208 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":91,"threads_started":1}
I20260811 11:53:11.251792 28957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:11.251333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.66:38085)
0811 11:53:11.251623 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:11.251683 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":79,"threads_started":1}
I20260811 11:53:11.470554 28958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:11.470062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.67:45313)
0811 11:53:11.470365 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:11.470431 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":81,"threads_started":1}
I20260811 11:53:11.808918 28959 raft_consensus.cc:482] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:53:11.809080 28959 raft_consensus.cc:504] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:11.809593 28959 leader_election.cc:288] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 903b03d331eb406da01ced709713f1fe (127.3.221.66:41217), 0efb25eba2184dbfb0e476a285fef172 (127.3.221.67:37931)
W20260811 11:53:11.810491 28272 leader_election.cc:341] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 903b03d331eb406da01ced709713f1fe (127.3.221.66:41217): Not found: Tablet not found: bd5bfe2caad84c1789e0ef6f160920dd
W20260811 11:53:11.810596 28275 leader_election.cc:341] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0efb25eba2184dbfb0e476a285fef172 (127.3.221.67:37931): Not found: Tablet not found: bd5bfe2caad84c1789e0ef6f160920dd
I20260811 11:53:11.810678 28275 leader_election.cc:302] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc6b1d1884454450ae1f9cbc66519a24; no voters: 0efb25eba2184dbfb0e476a285fef172, 903b03d331eb406da01ced709713f1fe
I20260811 11:53:11.810834 28959 raft_consensus.cc:2726] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:53:12.123215 28960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:12.122585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.65:37633)
0811 11:53:12.122997 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:12.123082 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":97,"threads_started":1}
I20260811 11:53:12.253151 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:12.252679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.66:39521)
0811 11:53:12.252980 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:12.253044 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":101,"threads_started":1}
I20260811 11:53:12.472126 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:12.471605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.67:50787)
0811 11:53:12.471946 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:12.472010 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
I20260811 11:53:13.125067 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:13.124580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.65:44937)
0811 11:53:13.124900 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:13.124957 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":100,"threads_started":1}
I20260811 11:53:13.254591 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:13.254092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.66:51529)
0811 11:53:13.254420 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:13.254484 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":112,"threads_started":1}
I20260811 11:53:13.473565 28967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:13.472997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.67:33207)
0811 11:53:13.473364 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:13.473462 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":90,"threads_started":1}
I20260811 11:53:14.126324 28969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:14.125856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.65:49819)
0811 11:53:14.126166 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:14.126227 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":113,"threads_started":1}
I20260811 11:53:14.256052 28970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:14.255520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.66:56025)
0811 11:53:14.255839 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:14.255917 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":91,"threads_started":1}
I20260811 11:53:14.475126 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:14.474547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.67:45951)
0811 11:53:14.474920 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:14.475000 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":92,"threads_started":1}
I20260811 11:53:15.127601 28974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:15.127129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.65:53091)
0811 11:53:15.127437 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:15.127501 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":81,"threads_started":1}
I20260811 11:53:15.183924 28975 raft_consensus.cc:482] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:53:15.184077 28975 raft_consensus.cc:504] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:15.184520 28975 leader_election.cc:288] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 903b03d331eb406da01ced709713f1fe (127.3.221.66:41217), 0efb25eba2184dbfb0e476a285fef172 (127.3.221.67:37931)
W20260811 11:53:15.185555 28272 leader_election.cc:341] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 903b03d331eb406da01ced709713f1fe (127.3.221.66:41217): Not found: Tablet not found: bd5bfe2caad84c1789e0ef6f160920dd
W20260811 11:53:15.185879 28275 leader_election.cc:341] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0efb25eba2184dbfb0e476a285fef172 (127.3.221.67:37931): Not found: Tablet not found: bd5bfe2caad84c1789e0ef6f160920dd
I20260811 11:53:15.186097 28275 leader_election.cc:302] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc6b1d1884454450ae1f9cbc66519a24; no voters: 0efb25eba2184dbfb0e476a285fef172, 903b03d331eb406da01ced709713f1fe
I20260811 11:53:15.186357 28975 raft_consensus.cc:2726] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:53:15.257526 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:15.256972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.66:41461)
0811 11:53:15.257368 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:15.257422 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":96,"threads_started":1}
I20260811 11:53:15.476583 28980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:15.476015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.67:37209)
0811 11:53:15.476419 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:15.476483 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":154,"threads_started":1}
I20260811 11:53:16.129614 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:16.128751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.65:39389)
0811 11:53:16.129422 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:16.129483 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":100,"threads_started":1}
I20260811 11:53:16.258881 28990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:16.258421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.66:46263)
0811 11:53:16.258709 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:16.258764 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":81,"threads_started":1}
I20260811 11:53:16.478030 28993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:16.477493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.67:51627)
0811 11:53:16.477853 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:16.477917 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":159,"threads_started":1}
I20260811 11:53:17.131197 28995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:17.130668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.65:47273)
0811 11:53:17.131025 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:17.131092 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":124,"threads_started":1}
I20260811 11:53:17.260331 28996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:17.259823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.66:35187)
0811 11:53:17.260149 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:17.260221 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":86,"threads_started":1}
I20260811 11:53:17.313858 28998 raft_consensus.cc:482] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:53:17.314020 28998 raft_consensus.cc:504] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:17.314494 28998 leader_election.cc:288] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 903b03d331eb406da01ced709713f1fe (127.3.221.66:41217), 0efb25eba2184dbfb0e476a285fef172 (127.3.221.67:37931)
W20260811 11:53:17.315623 28275 leader_election.cc:341] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0efb25eba2184dbfb0e476a285fef172 (127.3.221.67:37931): Not found: Tablet not found: bd5bfe2caad84c1789e0ef6f160920dd
W20260811 11:53:17.315752 28272 leader_election.cc:341] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 903b03d331eb406da01ced709713f1fe (127.3.221.66:41217): Not found: Tablet not found: bd5bfe2caad84c1789e0ef6f160920dd
I20260811 11:53:17.315840 28272 leader_election.cc:302] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc6b1d1884454450ae1f9cbc66519a24; no voters: 0efb25eba2184dbfb0e476a285fef172, 903b03d331eb406da01ced709713f1fe
I20260811 11:53:17.316020 28998 raft_consensus.cc:2726] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:53:17.479295 28999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:17.478834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.67:49521)
0811 11:53:17.479139 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:17.479195 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":117,"threads_started":1}
I20260811 11:53:18.132997 29000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:18.132451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.65:40807)
0811 11:53:18.132795 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:18.132875 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":89,"threads_started":1}
I20260811 11:53:18.261801 29002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:18.261284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.66:43707)
0811 11:53:18.261625 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:18.261689 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":99,"threads_started":1}
I20260811 11:53:18.480863 29005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:18.480294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.67:60051)
0811 11:53:18.480659 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:18.480732 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":125,"threads_started":1}
I20260811 11:53:19.134346 29007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:19.133830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.65:56239)
0811 11:53:19.134167 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:19.134237 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":91,"threads_started":1}
I20260811 11:53:19.263103 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:19.262630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.66:54403)
0811 11:53:19.262941 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:19.263002 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":96,"threads_started":1}
I20260811 11:53:19.482390 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:19.481882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.67:40053)
0811 11:53:19.482212 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:19.482283 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":109,"threads_started":1}
I20260811 11:53:20.135731 29012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:20.135245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.65:37455)
0811 11:53:20.135566 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:20.135630 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":88,"threads_started":1}
I20260811 11:53:20.264333 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:20.263897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.66:60893)
0811 11:53:20.264170 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:20.264224 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":85,"threads_started":1}
I20260811 11:53:20.483848 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:20.483241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.67:45787)
0811 11:53:20.483633 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:20.483699 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":110,"threads_started":1}
I20260811 11:53:20.841358 29017 raft_consensus.cc:482] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:53:20.841504 29017 raft_consensus.cc:504] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903b03d331eb406da01ced709713f1fe" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 41217 } } peers { permanent_uuid: "0efb25eba2184dbfb0e476a285fef172" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37931 } } peers { permanent_uuid: "bc6b1d1884454450ae1f9cbc66519a24" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 32839 } }
I20260811 11:53:20.841957 29017 leader_election.cc:288] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 903b03d331eb406da01ced709713f1fe (127.3.221.66:41217), 0efb25eba2184dbfb0e476a285fef172 (127.3.221.67:37931)
W20260811 11:53:20.842792 28272 leader_election.cc:341] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 903b03d331eb406da01ced709713f1fe (127.3.221.66:41217): Not found: Tablet not found: bd5bfe2caad84c1789e0ef6f160920dd
W20260811 11:53:20.842908 28275 leader_election.cc:341] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0efb25eba2184dbfb0e476a285fef172 (127.3.221.67:37931): Not found: Tablet not found: bd5bfe2caad84c1789e0ef6f160920dd
I20260811 11:53:20.842971 28275 leader_election.cc:302] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc6b1d1884454450ae1f9cbc66519a24; no voters: 0efb25eba2184dbfb0e476a285fef172, 903b03d331eb406da01ced709713f1fe
I20260811 11:53:20.843174 29017 raft_consensus.cc:2726] T bd5bfe2caad84c1789e0ef6f160920dd P bc6b1d1884454450ae1f9cbc66519a24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:53:21.137285 29018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:21.136749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.65:47433)
0811 11:53:21.137078 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:21.137143 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":88,"threads_started":1}
I20260811 11:53:21.265810 29021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:21.265266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.66:34767)
0811 11:53:21.265616 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:21.265680 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":161,"threads_started":1}
I20260811 11:53:21.485759 29024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:21.485079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.67:43043)
0811 11:53:21.485551 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:21.485644 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":146,"threads_started":1}
I20260811 11:53:22.138613 29025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:22.138150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.65:45725)
0811 11:53:22.138455 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:22.138512 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":113,"threads_started":1}
I20260811 11:53:22.267181 29026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:22.266708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.66:57271)
0811 11:53:22.267012 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:22.267077 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":92,"threads_started":1}
I20260811 11:53:22.487425 29027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:22.486867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.67:39937)
0811 11:53:22.487238 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:22.487303 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":123,"threads_started":1}
I20260811 11:53:23.140218 29028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:23.139734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.65:37981)
0811 11:53:23.140055 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:23.140118 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":89,"threads_started":1}
I20260811 11:53:23.268648 29029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:23.268172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.66:50393)
0811 11:53:23.268482 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:23.268545 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":90,"threads_started":1}
I20260811 11:53:23.477380 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 28188
I20260811 11:53:23.490731 29030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:23.489621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.67:39683)
0811 11:53:23.490557 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:23.490621 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":857,"thread_start_us":662,"threads_started":1}
I20260811 11:53:23.503683 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 28489
I20260811 11:53:23.527825 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 28713
2026-08-11T11:53:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (20259 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:53:23 dist-test-slave-bndk krb5kdc[29033](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:53:23 dist-test-slave-bndk krb5kdc[29033](info): set up 2 sockets
Aug 11 11:53:23 dist-test-slave-bndk krb5kdc[29033](info): commencing operation
krb5kdc: starting...
W20260811 11:53:24.748314 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:53:24 dist-test-slave-bndk krb5kdc[29033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:53:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:53:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:53:24.889436 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:36175
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:36175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:53:25.016482 29050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:53:25.016811 29050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:53:25.016875 29050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:53:25.021147 29050 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 11:53:25.021255 29050 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 11:53:25.021287 29050 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 11:53:25.021306 29050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:53:25.021340 29050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:53:25.021375 29050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:53:25.021407 29050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:53:25.021436 29050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:53:25.027184 29050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:36175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:36175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:53:25.028695 29050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:53:25.030370 29050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:53:25.040271 29058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:25.040310 29060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:25.040445 29050 server_base.cc:1034] running on GCE node
W20260811 11:53:25.040364 29064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:25.041285 29050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:53:25.042500 29050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:53:25.043684 29050 hybrid_clock.cc:648] HybridClock initialized: now 1786449205043668 us; error 32 us; skew 500 ppm
Aug 11 11:53:25 dist-test-slave-bndk krb5kdc[29033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449205, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:53:25.047325 29050 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:53:25.048463 29050 webserver.cc:460] Webserver started at http://127.3.221.126:36357/ using document root <none> and password file <none>
I20260811 11:53:25.049067 29050 fs_manager.cc:359] Metadata directory not provided
I20260811 11:53:25.049139 29050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:53:25.049434 29050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:53:25.051991 29050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "9a64185803e64ad6b861206d82019fd3"
format_stamp: "Formatted at 2026-08-11 11:53:25 on dist-test-slave-bndk"
server_key: "b97048684df170465f45c4105c3e3925"
server_key_iv: "66b57707dccea05cbbee72fa6a93c3c3"
server_key_version: "encryptionkey@0"
I20260811 11:53:25.052631 29050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "9a64185803e64ad6b861206d82019fd3"
format_stamp: "Formatted at 2026-08-11 11:53:25 on dist-test-slave-bndk"
server_key: "b97048684df170465f45c4105c3e3925"
server_key_iv: "66b57707dccea05cbbee72fa6a93c3c3"
server_key_version: "encryptionkey@0"
I20260811 11:53:25.056994 29050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260811 11:53:25.059891 29076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:25.061473 29050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:53:25.061618 29050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "9a64185803e64ad6b861206d82019fd3"
format_stamp: "Formatted at 2026-08-11 11:53:25 on dist-test-slave-bndk"
server_key: "b97048684df170465f45c4105c3e3925"
server_key_iv: "66b57707dccea05cbbee72fa6a93c3c3"
server_key_version: "encryptionkey@0"
I20260811 11:53:25.061738 29050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:53:25.086688 29050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:53:25.090495 29050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:53:25.090706 29050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:53:25.098663 29050 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:36175
I20260811 11:53:25.098970 29165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:36175 every 8 connection(s)
I20260811 11:53:25.099731 29050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:53:25.102783 29166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:53:25.108534 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 29050
I20260811 11:53:25.108670 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:53:25.108974 3957 external_mini_cluster.cc:1428] Setting key 935a624267db5a6c756fee3a7614130f
I20260811 11:53:25.110536 29166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3: Bootstrap starting.
I20260811 11:53:25.113039 29166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3: Neither blocks nor log segments found. Creating new log.
I20260811 11:53:25.114010 29166 log.cc:824] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3: Log is configured to *not* fsync() on all Append() calls
I20260811 11:53:25.116452 29166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3: No bootstrap required, opened a new log
Aug 11 11:53:25 dist-test-slave-bndk krb5kdc[29033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:53:25.121119 29166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a64185803e64ad6b861206d82019fd3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36175 } }
I20260811 11:53:25.121426 29166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:53:25.121487 29166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a64185803e64ad6b861206d82019fd3, State: Initialized, Role: FOLLOWER
I20260811 11:53:25.122005 29166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a64185803e64ad6b861206d82019fd3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36175 } }
I20260811 11:53:25.122187 29166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:53:25.122262 29166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:53:25.122375 29166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:53:25.123760 29166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a64185803e64ad6b861206d82019fd3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36175 } }
I20260811 11:53:25.124166 29166 leader_election.cc:302] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a64185803e64ad6b861206d82019fd3; no voters:
I20260811 11:53:25.124881 29166 leader_election.cc:288] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:53:25.125146 29173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:53:25.125454 29173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 1 LEADER]: Becoming Leader. State: Replica: 9a64185803e64ad6b861206d82019fd3, State: Running, Role: LEADER
I20260811 11:53:25.125906 29173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a64185803e64ad6b861206d82019fd3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36175 } }
I20260811 11:53:25.126642 29166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:53:25.127588 29171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:25.110530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46794 (local address 127.3.221.126:36175)
0811 11:53:25.110986 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:25.111003 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:25.111041 (+ 38us) server_negotiation.cc:407] Connection header received
0811 11:53:25.112140 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:25.112164 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:25.112538 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:25.112841 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:25.113800 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.115043 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.116251 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.116817 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.116956 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:25.119999 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:25.120005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:25.120023 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:25.120058 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:25.121905 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:25.122482 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:25.122489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:25.122496 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:25.122632 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:25.123007 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:25.123011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:25.123012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:25.124127 (+ 1115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:25.124508 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:25.125614 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0811 11:53:25.125997 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":144,"threads_started":1}
I20260811 11:53:25.128358 29176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a64185803e64ad6b861206d82019fd3. Latest consensus state: current_term: 1 leader_uuid: "9a64185803e64ad6b861206d82019fd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a64185803e64ad6b861206d82019fd3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36175 } } }
I20260811 11:53:25.128505 29176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [sys.catalog]: This master's current role is: LEADER
I20260811 11:53:25.128985 29174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a64185803e64ad6b861206d82019fd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a64185803e64ad6b861206d82019fd3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36175 } } }
I20260811 11:53:25.129240 29174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [sys.catalog]: This master's current role is: LEADER
I20260811 11:53:25.131942 29189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:53:25.135452 29189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:53:25.142364 29189 catalog_manager.cc:1347] Generated new cluster ID: dd85785dea4a43dfbe9da43ed0c7f4dd
I20260811 11:53:25.142462 29189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:53:25.164924 29189 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:53:25.166484 29189 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:53:25.177927 29189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3: Generated new TSK 0
I20260811 11:53:25.178920 29189 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
I20260811 11:53:25.212320 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36175
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:53:25.336638 29206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:53:25.336966 29206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:53:25.337014 29206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:53:25.341506 29206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:53:25.341588 29206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:53:25.341650 29206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:53:25.347399 29206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:53:25.349071 29206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:53:25.350711 29206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:53:25.362087 29215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:25.362087 29213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:25.362082 29212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:25.362556 29206 server_base.cc:1034] running on GCE node
I20260811 11:53:25.362998 29206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:53:25.364037 29206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:53:25.365516 29206 hybrid_clock.cc:648] HybridClock initialized: now 1786449205365437 us; error 53 us; skew 500 ppm
Aug 11 11:53:25 dist-test-slave-bndk krb5kdc[29033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449205, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:53:25.368922 29206 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:53:25.370107 29206 webserver.cc:460] Webserver started at http://127.3.221.65:45867/ using document root <none> and password file <none>
I20260811 11:53:25.370725 29206 fs_manager.cc:359] Metadata directory not provided
I20260811 11:53:25.370803 29206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:53:25.371011 29206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:53:25.373310 29206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "87a977b6ceef49e988b94b6d5c3215fe"
format_stamp: "Formatted at 2026-08-11 11:53:25 on dist-test-slave-bndk"
server_key: "526e1c04fb107212ed2f7922088fdc89"
server_key_iv: "c5f6d42994686adb4bd9d8bace33227d"
server_key_version: "encryptionkey@0"
I20260811 11:53:25.373932 29206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "87a977b6ceef49e988b94b6d5c3215fe"
format_stamp: "Formatted at 2026-08-11 11:53:25 on dist-test-slave-bndk"
server_key: "526e1c04fb107212ed2f7922088fdc89"
server_key_iv: "c5f6d42994686adb4bd9d8bace33227d"
server_key_version: "encryptionkey@0"
I20260811 11:53:25.378163 29206 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 11:53:25.381064 29224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:25.382717 29206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 11:53:25.382854 29206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "87a977b6ceef49e988b94b6d5c3215fe"
format_stamp: "Formatted at 2026-08-11 11:53:25 on dist-test-slave-bndk"
server_key: "526e1c04fb107212ed2f7922088fdc89"
server_key_iv: "c5f6d42994686adb4bd9d8bace33227d"
server_key_version: "encryptionkey@0"
I20260811 11:53:25.383008 29206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:53:25.399170 29206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:53:25.402824 29206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:53:25.403003 29206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:53:25.403606 29206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:53:25.404590 29206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:53:25.404641 29206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:25.404685 29206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:53:25.404718 29206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:25.414543 29206 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:40733
I20260811 11:53:25.414623 29417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:40733 every 8 connection(s)
I20260811 11:53:25.415555 29206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:53:25.421260 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 29206
I20260811 11:53:25.421412 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:53:25.421751 3957 external_mini_cluster.cc:1428] Setting key 7844362ed13a5838c705530822a5f6a3
Aug 11 11:53:25 dist-test-slave-bndk krb5kdc[29033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449205, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:53:25.432341 29171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:25.417631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:34789 (local address 127.3.221.126:36175)
0811 11:53:25.417786 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:25.417790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:25.418991 (+ 1201us) server_negotiation.cc:407] Connection header received
0811 11:53:25.419951 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:25.419955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:25.420016 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:25.420095 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:25.421749 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.422316 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.423865 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.424103 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.424194 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:25.427406 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:25.427410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:25.427413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:25.427444 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:25.429008 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:25.429757 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:25.429761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:25.429763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:25.429864 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:25.430205 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:25.430208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:25.430210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:25.430475 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:25.430609 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:25.431328 (+ 719us) server_negotiation.cc:299] Negotiation successful
0811 11:53:25.431982 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 11:53:25.432732 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:25.418004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.65:34789)
0811 11:53:25.418819 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:25.418865 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:25.419726 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:25.420306 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:25.420314 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:25.420784 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:53:25.421544 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:25.421568 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.422461 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.422465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:25.423710 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:25.423723 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.424294 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.424298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:25.424504 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:25.425153 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:53:25.425176 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:53:25.427156 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:53:25.429167 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:25.429175 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:25.429190 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:25.429613 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:25.429951 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:25.429953 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:25.429955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:25.430099 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:25.430666 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:53:25.430729 (+ 63us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:53:25.431041 (+ 312us) client_negotiation.cc:769] Sending connection context
0811 11:53:25.431233 (+ 192us) client_negotiation.cc:240] Negotiation successful
0811 11:53:25.431744 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":138,"threads_started":1}
I20260811 11:53:25.434204 29418 heartbeater.cc:347] Connected to a master server at 127.3.221.126:36175
I20260811 11:53:25.434506 29418 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.221.66@KRBTEST.COM; defaulting to no policy
I20260811 11:53:25.435074 29418 heartbeater.cc:511] Master 127.3.221.126:36175 requested a full tablet report, sending...
I20260811 11:53:25.437546 29102 ts_manager.cc:194] Registered new tserver with Master: 87a977b6ceef49e988b94b6d5c3215fe (127.3.221.65:40733)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.66@KRBTEST.COM" created.
I20260811 11:53:25.438894 29102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:34789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
I20260811 11:53:25.456578 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36175
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:53:25.580812 29429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:53:25.581127 29429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:53:25.581192 29429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:53:25.585392 29429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:53:25.585469 29429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:53:25.585551 29429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:53:25.591521 29429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:53:25.592747 29429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:53:25.594018 29429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:53:25.604503 29444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:25.604568 29442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:25.604715 29429 server_base.cc:1034] running on GCE node
W20260811 11:53:25.604617 29440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:25.605366 29429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:53:25.606081 29429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:53:25.607268 29429 hybrid_clock.cc:648] HybridClock initialized: now 1786449205607234 us; error 47 us; skew 500 ppm
Aug 11 11:53:25 dist-test-slave-bndk krb5kdc[29033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449205, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:53:25.610702 29429 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:53:25.611688 29429 webserver.cc:460] Webserver started at http://127.3.221.66:36053/ using document root <none> and password file <none>
I20260811 11:53:25.612298 29429 fs_manager.cc:359] Metadata directory not provided
I20260811 11:53:25.612368 29429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:53:25.612550 29429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:53:25.614989 29429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "0642a3ad98fe46869269ca187bd2d5fd"
format_stamp: "Formatted at 2026-08-11 11:53:25 on dist-test-slave-bndk"
server_key: "d9695daf9862f39a7446a81ef5768419"
server_key_iv: "b77e0c71a86320dba523b9ae71d57ca0"
server_key_version: "encryptionkey@0"
I20260811 11:53:25.615599 29429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "0642a3ad98fe46869269ca187bd2d5fd"
format_stamp: "Formatted at 2026-08-11 11:53:25 on dist-test-slave-bndk"
server_key: "d9695daf9862f39a7446a81ef5768419"
server_key_iv: "b77e0c71a86320dba523b9ae71d57ca0"
server_key_version: "encryptionkey@0"
I20260811 11:53:25.619632 29429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 11:53:25.622289 29455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:25.623636 29429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:53:25.623782 29429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "0642a3ad98fe46869269ca187bd2d5fd"
format_stamp: "Formatted at 2026-08-11 11:53:25 on dist-test-slave-bndk"
server_key: "d9695daf9862f39a7446a81ef5768419"
server_key_iv: "b77e0c71a86320dba523b9ae71d57ca0"
server_key_version: "encryptionkey@0"
I20260811 11:53:25.623900 29429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:53:25.644397 29429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:53:25.647816 29429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:53:25.647997 29429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:53:25.648600 29429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:53:25.649531 29429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:53:25.649585 29429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:25.649626 29429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:53:25.649639 29429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:25.658576 29429 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:38893
I20260811 11:53:25.658600 29640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:38893 every 8 connection(s)
I20260811 11:53:25.659538 29429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:53:25.665083 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 29429
I20260811 11:53:25.665243 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:53:25.665551 3957 external_mini_cluster.cc:1428] Setting key f3437785b248d9b05e6c8234df5cae33
Aug 11 11:53:25 dist-test-slave-bndk krb5kdc[29033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449205, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:53:25.673631 29171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:25.661431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:38867 (local address 127.3.221.126:36175)
0811 11:53:25.661558 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:25.661562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:25.662428 (+ 866us) server_negotiation.cc:407] Connection header received
0811 11:53:25.663742 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:25.663746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:25.663805 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:25.663880 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:25.665297 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.665910 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.667406 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.667626 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.667714 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:25.670227 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:25.670231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:25.670233 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:25.670256 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:25.671553 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:25.672067 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:25.672070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:25.672071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:25.672147 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:25.672475 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:25.672478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:25.672480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:25.672671 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:25.672758 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:25.673323 (+ 565us) server_negotiation.cc:299] Negotiation successful
0811 11:53:25.673458 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 11:53:25.674494 29644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:25.661799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.66:38867)
0811 11:53:25.662252 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:25.662296 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:25.663532 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:25.664075 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:25.664082 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:25.664459 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:53:25.665083 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:25.665106 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.666056 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.666059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:25.667281 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:25.667293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.667778 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.667782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:25.667950 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:25.668519 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:53:25.668542 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:53:25.670052 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:53:25.671671 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:25.671678 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:25.671695 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:25.671969 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:25.672232 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:25.672235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:25.672237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:25.672383 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:25.672782 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:53:25.672789 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:53:25.673047 (+ 258us) client_negotiation.cc:769] Sending connection context
0811 11:53:25.673250 (+ 203us) client_negotiation.cc:240] Negotiation successful
0811 11:53:25.673603 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.3.221.67@KRBTEST.COM; defaulting to no policy
I20260811 11:53:25.675792 29641 heartbeater.cc:347] Connected to a master server at 127.3.221.126:36175
I20260811 11:53:25.676080 29641 heartbeater.cc:464] Registering TS with master...
I20260811 11:53:25.676644 29641 heartbeater.cc:511] Master 127.3.221.126:36175 requested a full tablet report, sending...
I20260811 11:53:25.677829 29102 ts_manager.cc:194] Registered new tserver with Master: 0642a3ad98fe46869269ca187bd2d5fd (127.3.221.66:38893)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.67@KRBTEST.COM" created.
I20260811 11:53:25.678511 29102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:38867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
I20260811 11:53:25.696192 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36175
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:53:25.818215 29647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:53:25.818507 29647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:53:25.818552 29647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:53:25.822611 29647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:53:25.822683 29647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:53:25.822743 29647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:53:25.828336 29647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:53:25.829523 29647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:53:25.830753 29647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:53:25.841157 29655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:25.841239 29659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:25.841312 29647 server_base.cc:1034] running on GCE node
W20260811 11:53:25.841244 29657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:25.841871 29647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:53:25.842588 29647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:53:25.843740 29647 hybrid_clock.cc:648] HybridClock initialized: now 1786449205843705 us; error 47 us; skew 500 ppm
Aug 11 11:53:25 dist-test-slave-bndk krb5kdc[29033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449205, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:53:25.846655 29647 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:53:25.847754 29647 webserver.cc:460] Webserver started at http://127.3.221.67:39321/ using document root <none> and password file <none>
I20260811 11:53:25.848335 29647 fs_manager.cc:359] Metadata directory not provided
I20260811 11:53:25.848405 29647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:53:25.848662 29647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:53:25.850950 29647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "dc6745c5de9b426690d9e3ec5ae80bb9"
format_stamp: "Formatted at 2026-08-11 11:53:25 on dist-test-slave-bndk"
server_key: "23dbd358c7ce80ac09e529116902bb89"
server_key_iv: "c8b93a517fd4bbcdf50f1a4c2ff8e07f"
server_key_version: "encryptionkey@0"
I20260811 11:53:25.851514 29647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "dc6745c5de9b426690d9e3ec5ae80bb9"
format_stamp: "Formatted at 2026-08-11 11:53:25 on dist-test-slave-bndk"
server_key: "23dbd358c7ce80ac09e529116902bb89"
server_key_iv: "c8b93a517fd4bbcdf50f1a4c2ff8e07f"
server_key_version: "encryptionkey@0"
I20260811 11:53:25.855566 29647 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 11:53:25.858498 29675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:25.860080 29647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:53:25.860208 29647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "dc6745c5de9b426690d9e3ec5ae80bb9"
format_stamp: "Formatted at 2026-08-11 11:53:25 on dist-test-slave-bndk"
server_key: "23dbd358c7ce80ac09e529116902bb89"
server_key_iv: "c8b93a517fd4bbcdf50f1a4c2ff8e07f"
server_key_version: "encryptionkey@0"
I20260811 11:53:25.860329 29647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:53:25.882324 29647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:53:25.885689 29647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:53:25.885865 29647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:53:25.886456 29647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:53:25.887359 29647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:53:25.887409 29647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:25.887451 29647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:53:25.887470 29647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:25.896642 29647 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:46597
I20260811 11:53:25.896688 29849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:46597 every 8 connection(s)
I20260811 11:53:25.897637 29647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:53:25.905243 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 29647
I20260811 11:53:25.905392 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:53:25.905682 3957 external_mini_cluster.cc:1428] Setting key 09f1f972ede4aa8623cf033b432891a3
Aug 11 11:53:25 dist-test-slave-bndk krb5kdc[29033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449205, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:53:25.913744 29171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:25.899747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:52271 (local address 127.3.221.126:36175)
0811 11:53:25.899891 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:25.899895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:25.901513 (+ 1618us) server_negotiation.cc:407] Connection header received
0811 11:53:25.902494 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:25.902497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:25.902551 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:25.902638 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:25.904601 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.905330 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.907038 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.907284 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.907374 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:25.909921 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:25.909925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:25.909927 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:25.909953 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:25.911487 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:25.912045 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:25.912048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:25.912050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:25.912133 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:25.912486 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:25.912490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:25.912492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:25.912698 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:25.912786 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:25.913424 (+ 638us) server_negotiation.cc:299] Negotiation successful
0811 11:53:25.913572 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 11:53:25.914625 29852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:25.900131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.67:52271)
0811 11:53:25.901194 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:25.901397 (+ 203us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:25.902232 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:25.903008 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:25.903019 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:25.903741 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:53:25.904303 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:25.904330 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.905634 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.905637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:25.906902 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:25.906913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.907435 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.907439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:25.907623 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:25.908198 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:53:25.908216 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:53:25.909740 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:53:25.911624 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:25.911631 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:25.911650 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:25.911943 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:25.912221 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:25.912224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:25.912226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:25.912389 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:25.912806 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:53:25.912813 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:53:25.913131 (+ 318us) client_negotiation.cc:769] Sending connection context
0811 11:53:25.913361 (+ 230us) client_negotiation.cc:240] Negotiation successful
0811 11:53:25.913724 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":128,"threads_started":1}
I20260811 11:53:25.915940 29850 heartbeater.cc:347] Connected to a master server at 127.3.221.126:36175
I20260811 11:53:25.916217 29850 heartbeater.cc:464] Registering TS with master...
I20260811 11:53:25.916790 29850 heartbeater.cc:511] Master 127.3.221.126:36175 requested a full tablet report, sending...
I20260811 11:53:25.917971 29102 ts_manager.cc:194] Registered new tserver with Master: dc6745c5de9b426690d9e3ec5ae80bb9 (127.3.221.67:46597)
I20260811 11:53:25.918642 29102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:52271
I20260811 11:53:25.920274 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:53:25.931109 29171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:25.922486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46804 (local address 127.3.221.126:36175)
0811 11:53:25.922626 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:25.922630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:25.922814 (+ 184us) server_negotiation.cc:407] Connection header received
0811 11:53:25.922863 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:25.922865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:25.922910 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:25.922984 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:25.923717 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.924251 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.925300 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.925619 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.925793 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:25.927859 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:25.927863 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:25.927866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:25.927908 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:25.929450 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:25.929872 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:25.929875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:25.929876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:25.929948 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:25.930143 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:25.930145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:25.930147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:25.930403 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:25.930534 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:25.930805 (+ 271us) server_negotiation.cc:299] Negotiation successful
0811 11:53:25.930943 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 11:53:25.934177 29101 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:53:25.937309 29101 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:53:25.952383 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:25.946052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:38893 (local address 127.0.0.1:36220)
0811 11:53:25.946942 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:25.946996 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:25.947151 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:25.948010 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:25.948016 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:25.948040 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:53:25.948190 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:25.948198 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.949537 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.949540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:25.950798 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:25.950810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.951824 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.951827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:25.951946 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:25.951987 (+ 41us) client_negotiation.cc:769] Sending connection context
0811 11:53:25.952098 (+ 111us) client_negotiation.cc:240] Negotiation successful
0811 11:53:25.952231 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":84,"threads_started":1}
I20260811 11:53:25.952396 29864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:25.946210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36220 (local address 127.3.221.66:38893)
0811 11:53:25.946577 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:25.946590 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:25.947453 (+ 863us) server_negotiation.cc:407] Connection header received
0811 11:53:25.947640 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:25.947647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:25.947848 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:25.948006 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:53:25.948483 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.949410 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.950905 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.951707 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.951811 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:25.951982 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:25.952139 (+ 157us) server_negotiation.cc:299] Negotiation successful
0811 11:53:25.952241 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":132,"threads_started":1}
I20260811 11:53:25.952795 29863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:25.946139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:40733 (local address 127.0.0.1:37568)
0811 11:53:25.947173 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:25.947187 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:25.947276 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:25.948372 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:25.948375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:25.948394 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:53:25.948505 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:25.948512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.950133 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.950136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:25.951316 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:25.951330 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.952442 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.952445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:25.952527 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:25.952551 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 11:53:25.952595 (+ 44us) client_negotiation.cc:240] Negotiation successful
0811 11:53:25.952662 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"thread_start_us":96,"threads_started":1}
I20260811 11:53:25.953613 29865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:25.946760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37568 (local address 127.3.221.65:40733)
0811 11:53:25.947887 (+ 1127us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:25.947894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:25.947917 (+ 23us) server_negotiation.cc:407] Connection header received
0811 11:53:25.947992 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:25.947998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:25.948205 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:25.948658 (+ 453us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:53:25.948798 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.949979 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.951485 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.952295 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.953046 (+ 751us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:25.953313 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:25.953370 (+ 57us) server_negotiation.cc:299] Negotiation successful
0811 11:53:25.953472 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1022,"thread_start_us":135,"threads_started":1}
I20260811 11:53:25.954468 29867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:25.946806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:46597 (local address 127.0.0.1:45520)
0811 11:53:25.947826 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:25.947842 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:25.947955 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:25.948643 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:25.948646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:25.948666 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:53:25.948758 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:25.948765 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.950162 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.950165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:25.952436 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:25.952445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.954093 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.954097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:25.954182 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:25.954206 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 11:53:25.954275 (+ 69us) client_negotiation.cc:240] Negotiation successful
0811 11:53:25.954340 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"mutex_wait_us":1154,"thread_start_us":105,"threads_started":1}
I20260811 11:53:25.954952 29868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:25.947234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45520 (local address 127.3.221.67:46597)
0811 11:53:25.948219 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:25.948231 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:25.948254 (+ 23us) server_negotiation.cc:407] Connection header received
0811 11:53:25.948327 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:25.948333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:25.948512 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:25.948657 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:53:25.948878 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.950026 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.952886 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:25.953784 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:25.953888 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:25.954029 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:25.954707 (+ 678us) server_negotiation.cc:299] Negotiation successful
0811 11:53:25.954804 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":876,"thread_start_us":247,"threads_started":1}
I20260811 11:53:25.955968 29542 tablet_service.cc:1467] Processing CreateTablet for tablet 44ab24778d7140dda7a8e96f4effea3d (DEFAULT_TABLE table=test-table [id=e6bacb02d47c40b2ab2ee8e95b420708]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:53:25.956127 29307 tablet_service.cc:1467] Processing CreateTablet for tablet 44ab24778d7140dda7a8e96f4effea3d (DEFAULT_TABLE table=test-table [id=e6bacb02d47c40b2ab2ee8e95b420708]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:53:25.956930 29542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44ab24778d7140dda7a8e96f4effea3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:53:25.957199 29307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44ab24778d7140dda7a8e96f4effea3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:53:25.957629 29751 tablet_service.cc:1467] Processing CreateTablet for tablet 44ab24778d7140dda7a8e96f4effea3d (DEFAULT_TABLE table=test-table [id=e6bacb02d47c40b2ab2ee8e95b420708]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:53:25.958698 29751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44ab24778d7140dda7a8e96f4effea3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:53:25.963665 29870 tablet_bootstrap.cc:492] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd: Bootstrap starting.
I20260811 11:53:25.966428 29872 tablet_bootstrap.cc:492] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe: Bootstrap starting.
I20260811 11:53:25.967495 29870 tablet_bootstrap.cc:654] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd: Neither blocks nor log segments found. Creating new log.
I20260811 11:53:25.968271 29870 log.cc:824] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd: Log is configured to *not* fsync() on all Append() calls
I20260811 11:53:25.968433 29872 tablet_bootstrap.cc:654] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe: Neither blocks nor log segments found. Creating new log.
I20260811 11:53:25.968982 29874 tablet_bootstrap.cc:492] T 44ab24778d7140dda7a8e96f4effea3d P dc6745c5de9b426690d9e3ec5ae80bb9: Bootstrap starting.
I20260811 11:53:25.969285 29872 log.cc:824] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe: Log is configured to *not* fsync() on all Append() calls
I20260811 11:53:25.970495 29870 tablet_bootstrap.cc:492] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd: No bootstrap required, opened a new log
I20260811 11:53:25.970708 29870 ts_tablet_manager.cc:1397] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260811 11:53:25.971082 29874 tablet_bootstrap.cc:654] T 44ab24778d7140dda7a8e96f4effea3d P dc6745c5de9b426690d9e3ec5ae80bb9: Neither blocks nor log segments found. Creating new log.
I20260811 11:53:25.971372 29872 tablet_bootstrap.cc:492] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe: No bootstrap required, opened a new log
I20260811 11:53:25.971562 29872 ts_tablet_manager.cc:1397] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260811 11:53:25.972254 29874 log.cc:824] T 44ab24778d7140dda7a8e96f4effea3d P dc6745c5de9b426690d9e3ec5ae80bb9: Log is configured to *not* fsync() on all Append() calls
I20260811 11:53:25.974211 29874 tablet_bootstrap.cc:492] T 44ab24778d7140dda7a8e96f4effea3d P dc6745c5de9b426690d9e3ec5ae80bb9: No bootstrap required, opened a new log
I20260811 11:53:25.974414 29874 ts_tablet_manager.cc:1397] T 44ab24778d7140dda7a8e96f4effea3d P dc6745c5de9b426690d9e3ec5ae80bb9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 11:53:25.974850 29870 raft_consensus.cc:348] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } }
I20260811 11:53:25.975164 29870 raft_consensus.cc:374] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:53:25.975247 29870 raft_consensus.cc:729] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0642a3ad98fe46869269ca187bd2d5fd, State: Initialized, Role: FOLLOWER
I20260811 11:53:25.975764 29870 consensus_queue.cc:260] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } }
I20260811 11:53:25.976130 29872 raft_consensus.cc:348] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } }
I20260811 11:53:25.976478 29872 raft_consensus.cc:374] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:53:25.976555 29872 raft_consensus.cc:729] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87a977b6ceef49e988b94b6d5c3215fe, State: Initialized, Role: FOLLOWER
I20260811 11:53:25.976652 29641 heartbeater.cc:503] Master 127.3.221.126:36175 was elected leader, sending a full tablet report...
I20260811 11:53:25.977017 29870 ts_tablet_manager.cc:1428] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260811 11:53:25.977025 29872 consensus_queue.cc:260] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } }
I20260811 11:53:25.977890 29872 ts_tablet_manager.cc:1428] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260811 11:53:25.977860 29874 raft_consensus.cc:348] T 44ab24778d7140dda7a8e96f4effea3d P dc6745c5de9b426690d9e3ec5ae80bb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } }
I20260811 11:53:25.978120 29874 raft_consensus.cc:374] T 44ab24778d7140dda7a8e96f4effea3d P dc6745c5de9b426690d9e3ec5ae80bb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:53:25.978180 29874 raft_consensus.cc:729] T 44ab24778d7140dda7a8e96f4effea3d P dc6745c5de9b426690d9e3ec5ae80bb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc6745c5de9b426690d9e3ec5ae80bb9, State: Initialized, Role: FOLLOWER
I20260811 11:53:25.978228 29418 heartbeater.cc:503] Master 127.3.221.126:36175 was elected leader, sending a full tablet report...
I20260811 11:53:25.978618 29874 consensus_queue.cc:260] T 44ab24778d7140dda7a8e96f4effea3d P dc6745c5de9b426690d9e3ec5ae80bb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } }
I20260811 11:53:25.979734 29874 ts_tablet_manager.cc:1428] T 44ab24778d7140dda7a8e96f4effea3d P dc6745c5de9b426690d9e3ec5ae80bb9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260811 11:53:25.979828 29850 heartbeater.cc:503] Master 127.3.221.126:36175 was elected leader, sending a full tablet report...
I20260811 11:53:26.133549 29881 raft_consensus.cc:482] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:53:26.133778 29881 raft_consensus.cc:504] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } }
I20260811 11:53:26.134956 29881 leader_election.cc:288] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87a977b6ceef49e988b94b6d5c3215fe (127.3.221.65:40733), dc6745c5de9b426690d9e3ec5ae80bb9 (127.3.221.67:46597)
I20260811 11:53:26.138846 29865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:26.135424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:34889 (local address 127.3.221.65:40733)
0811 11:53:26.135588 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:26.135592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:26.135607 (+ 15us) server_negotiation.cc:407] Connection header received
0811 11:53:26.135662 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:26.135665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:26.135715 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:26.135797 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:53:26.136134 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:26.136661 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:26.137851 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:26.138253 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:26.138365 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:26.138424 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:26.138621 (+ 197us) server_negotiation.cc:299] Negotiation successful
0811 11:53:26.138699 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260811 11:53:26.138845 29868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:26.135361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:39247 (local address 127.3.221.67:46597)
0811 11:53:26.135516 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:26.135520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:26.135742 (+ 222us) server_negotiation.cc:407] Connection header received
0811 11:53:26.135858 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:26.135861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:26.135908 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:26.135992 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:53:26.136226 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:26.136715 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:26.137851 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:26.138254 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:26.138365 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:26.138430 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:26.138640 (+ 210us) server_negotiation.cc:299] Negotiation successful
0811 11:53:26.138712 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260811 11:53:26.138845 29644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:26.135177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:40733 (local address 127.3.221.66:34889)
0811 11:53:26.135307 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:26.135323 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:26.135427 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:26.135841 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:26.135844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:26.135870 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:53:26.136017 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:26.136026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:26.136770 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:26.136774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:26.137727 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:26.137738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:26.138365 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:26.138369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:26.138495 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:26.138526 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 11:53:26.138588 (+ 62us) client_negotiation.cc:240] Negotiation successful
0811 11:53:26.138711 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":72}
I20260811 11:53:26.138859 29886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:26.135271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:46597 (local address 127.3.221.66:39247)
0811 11:53:26.135648 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:26.135660 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:26.135764 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:26.136001 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:26.136004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:26.136018 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:53:26.136124 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:26.136131 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:26.136826 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:26.136828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:26.137723 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:26.137735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:26.138374 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:26.138378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:26.138496 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:26.138538 (+ 42us) client_negotiation.cc:769] Sending connection context
0811 11:53:26.138607 (+ 69us) client_negotiation.cc:240] Negotiation successful
0811 11:53:26.138699 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":27,"thread_start_us":126,"threads_started":1}
I20260811 11:53:26.139564 29339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ab24778d7140dda7a8e96f4effea3d" candidate_uuid: "0642a3ad98fe46869269ca187bd2d5fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87a977b6ceef49e988b94b6d5c3215fe" is_pre_election: true
I20260811 11:53:26.139580 29783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ab24778d7140dda7a8e96f4effea3d" candidate_uuid: "0642a3ad98fe46869269ca187bd2d5fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" is_pre_election: true
I20260811 11:53:26.139865 29339 raft_consensus.cc:2454] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0642a3ad98fe46869269ca187bd2d5fd in term 0.
I20260811 11:53:26.139865 29783 raft_consensus.cc:2454] T 44ab24778d7140dda7a8e96f4effea3d P dc6745c5de9b426690d9e3ec5ae80bb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0642a3ad98fe46869269ca187bd2d5fd in term 0.
I20260811 11:53:26.140360 29458 leader_election.cc:302] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0642a3ad98fe46869269ca187bd2d5fd, 87a977b6ceef49e988b94b6d5c3215fe; no voters:
I20260811 11:53:26.140687 29881 raft_consensus.cc:2781] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:53:26.140781 29881 raft_consensus.cc:482] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:53:26.140812 29881 raft_consensus.cc:3037] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:53:26.141937 29881 raft_consensus.cc:504] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } }
I20260811 11:53:26.142323 29881 leader_election.cc:288] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd [CANDIDATE]: Term 1 election: Requested vote from peers 87a977b6ceef49e988b94b6d5c3215fe (127.3.221.65:40733), dc6745c5de9b426690d9e3ec5ae80bb9 (127.3.221.67:46597)
I20260811 11:53:26.142699 29783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ab24778d7140dda7a8e96f4effea3d" candidate_uuid: "0642a3ad98fe46869269ca187bd2d5fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9"
I20260811 11:53:26.142684 29339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ab24778d7140dda7a8e96f4effea3d" candidate_uuid: "0642a3ad98fe46869269ca187bd2d5fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87a977b6ceef49e988b94b6d5c3215fe"
I20260811 11:53:26.142841 29339 raft_consensus.cc:3037] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:53:26.142843 29783 raft_consensus.cc:3037] T 44ab24778d7140dda7a8e96f4effea3d P dc6745c5de9b426690d9e3ec5ae80bb9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:53:26.144186 29339 raft_consensus.cc:2454] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0642a3ad98fe46869269ca187bd2d5fd in term 1.
I20260811 11:53:26.144186 29783 raft_consensus.cc:2454] T 44ab24778d7140dda7a8e96f4effea3d P dc6745c5de9b426690d9e3ec5ae80bb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0642a3ad98fe46869269ca187bd2d5fd in term 1.
I20260811 11:53:26.144527 29458 leader_election.cc:302] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0642a3ad98fe46869269ca187bd2d5fd, 87a977b6ceef49e988b94b6d5c3215fe; no voters:
I20260811 11:53:26.144703 29881 raft_consensus.cc:2781] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:53:26.144941 29881 raft_consensus.cc:686] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd [term 1 LEADER]: Becoming Leader. State: Replica: 0642a3ad98fe46869269ca187bd2d5fd, State: Running, Role: LEADER
I20260811 11:53:26.145357 29881 consensus_queue.cc:237] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } }
W20260811 11:53:26.148833 29851 tablet.cc:2369] T 44ab24778d7140dda7a8e96f4effea3d P dc6745c5de9b426690d9e3ec5ae80bb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:53:26.149204 29102 catalog_manager.cc:5495] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 0642a3ad98fe46869269ca187bd2d5fd (127.3.221.66). New cstate: current_term: 1 leader_uuid: "0642a3ad98fe46869269ca187bd2d5fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } health_report { overall_health: UNKNOWN } } }
W20260811 11:53:26.160822 29643 tablet.cc:2369] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:53:26.167068 29420 tablet.cc:2369] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:53:26.197811 29864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:26.193987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36234 (local address 127.3.221.66:38893)
0811 11:53:26.194148 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:26.194152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:26.194190 (+ 38us) server_negotiation.cc:407] Connection header received
0811 11:53:26.194278 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:26.194280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:26.194355 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:26.194452 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:53:26.194712 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:26.195226 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:26.196125 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:26.196352 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:26.196441 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:26.196765 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:53:26.197258 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:53:26.197368 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:26.197529 (+ 161us) server_negotiation.cc:299] Negotiation successful
0811 11:53:26.197613 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 11:53:26.204593 29783 raft_consensus.cc:1261] T 44ab24778d7140dda7a8e96f4effea3d P dc6745c5de9b426690d9e3ec5ae80bb9 [term 1 FOLLOWER]: Refusing update from remote peer 0642a3ad98fe46869269ca187bd2d5fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:53:26.204593 29339 raft_consensus.cc:1261] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe [term 1 FOLLOWER]: Refusing update from remote peer 0642a3ad98fe46869269ca187bd2d5fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:53:26.205531 29888 consensus_queue.cc:1035] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:53:26.205763 29881 consensus_queue.cc:1035] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:53:26.214864 29898 mvcc.cc:205] Tried to move back new op lower bound from 7317295948604624896 to 7317295948374257664. Current Snapshot: MvccSnapshot[applied={T|T < 7317295948604624896}]
I20260811 11:53:26.214956 29896 mvcc.cc:205] Tried to move back new op lower bound from 7317295948604624896 to 7317295948374257664. Current Snapshot: MvccSnapshot[applied={T|T < 7317295948604624896}]
I20260811 11:53:26.224182 29102 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:53:26.224390 29102 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:53:26.227205 29102 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3: Sending DeleteTablet for 3 replicas of tablet 44ab24778d7140dda7a8e96f4effea3d
I20260811 11:53:26.228194 29542 tablet_service.cc:1514] Processing DeleteTablet for tablet 44ab24778d7140dda7a8e96f4effea3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:53:26 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:36220
I20260811 11:53:26.228242 29307 tablet_service.cc:1514] Processing DeleteTablet for tablet 44ab24778d7140dda7a8e96f4effea3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:53:26 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:37568
I20260811 11:53:26.228577 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 29206
I20260811 11:53:26.228827 29751 tablet_service.cc:1514] Processing DeleteTablet for tablet 44ab24778d7140dda7a8e96f4effea3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:53:26 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:45520
I20260811 11:53:26.235783 29916 tablet_replica.cc:330] T 44ab24778d7140dda7a8e96f4effea3d P dc6745c5de9b426690d9e3ec5ae80bb9: stopping tablet replica
I20260811 11:53:26.236263 29916 raft_consensus.cc:2229] T 44ab24778d7140dda7a8e96f4effea3d P dc6745c5de9b426690d9e3ec5ae80bb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:53:26.237104 29916 raft_consensus.cc:2258] T 44ab24778d7140dda7a8e96f4effea3d P dc6745c5de9b426690d9e3ec5ae80bb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:53:26.239343 29914 tablet_replica.cc:330] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd: stopping tablet replica
I20260811 11:53:26.239787 29914 raft_consensus.cc:2229] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:53:26.240329 29914 raft_consensus.cc:2258] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:53:26.242470 29916 ts_tablet_manager.cc:1905] T 44ab24778d7140dda7a8e96f4effea3d P dc6745c5de9b426690d9e3ec5ae80bb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:53:26.242846 29914 ts_tablet_manager.cc:1905] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:53:26.244851 29916 ts_tablet_manager.cc:1918] T 44ab24778d7140dda7a8e96f4effea3d P dc6745c5de9b426690d9e3ec5ae80bb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:53:26.244968 29916 log.cc:1196] T 44ab24778d7140dda7a8e96f4effea3d P dc6745c5de9b426690d9e3ec5ae80bb9: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/44ab24778d7140dda7a8e96f4effea3d
I20260811 11:53:26.245348 29916 ts_tablet_manager.cc:1939] T 44ab24778d7140dda7a8e96f4effea3d P dc6745c5de9b426690d9e3ec5ae80bb9: Deleting consensus metadata
I20260811 11:53:26.246905 29914 ts_tablet_manager.cc:1918] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:53:26.247028 29914 log.cc:1196] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/44ab24778d7140dda7a8e96f4effea3d
I20260811 11:53:26.247043 29084 catalog_manager.cc:4841] TS dc6745c5de9b426690d9e3ec5ae80bb9 (127.3.221.67:46597): tablet 44ab24778d7140dda7a8e96f4effea3d (table test-table [id=e6bacb02d47c40b2ab2ee8e95b420708]) successfully deleted
I20260811 11:53:26.247413 29914 ts_tablet_manager.cc:1939] T 44ab24778d7140dda7a8e96f4effea3d P 0642a3ad98fe46869269ca187bd2d5fd: Deleting consensus metadata
I20260811 11:53:26.248709 29080 catalog_manager.cc:4841] TS 0642a3ad98fe46869269ca187bd2d5fd (127.3.221.66:38893): tablet 44ab24778d7140dda7a8e96f4effea3d (table test-table [id=e6bacb02d47c40b2ab2ee8e95b420708]) successfully deleted
W20260811 11:53:26.251709 29077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.221.65:40733 (error 108)
I20260811 11:53:26.252745 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 29429
I20260811 11:53:26.252938 29867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:26.252607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:40733 (local address 127.0.0.1:37584)
0811 11:53:26.252755 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:26.252835 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.65:40733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260811 11:53:26.256330 29077 catalog_manager.cc:4584] TS 87a977b6ceef49e988b94b6d5c3215fe (127.3.221.65:40733): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 44ab24778d7140dda7a8e96f4effea3d: Network error: Client connection negotiation failed: client connection to 127.3.221.65:40733: connect: Connection refused (error 111)
I20260811 11:53:26.262776 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 29647
I20260811 11:53:26.281615 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 29050
I20260811 11:53:46.301043 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:36175
--webserver_interface=127.3.221.126
--webserver_port=36357
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:36175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:53:46.425697 29918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:53:46.426023 29918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:53:46.426090 29918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:53:46.430259 29918 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 11:53:46.430330 29918 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 11:53:46.430352 29918 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 11:53:46.430369 29918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:53:46.430393 29918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:53:46.430419 29918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:53:46.430449 29918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:53:46.430480 29918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:53:46.436623 29918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:36175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:36175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=36357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:53:46.437876 29918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:53:46.438951 29918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:53:46.455392 29926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:46.455325 29929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:46.455437 29927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:46.455399 29918 server_base.cc:1034] running on GCE node
I20260811 11:53:46.456573 29918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:53:46.457939 29918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:53:46.459133 29918 hybrid_clock.cc:648] HybridClock initialized: now 1786449226459102 us; error 52 us; skew 500 ppm
Aug 11 11:53:46 dist-test-slave-bndk krb5kdc[29033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449226, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:53:46.462859 29918 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:53:46.463994 29918 webserver.cc:460] Webserver started at http://127.3.221.126:36357/ using document root <none> and password file <none>
I20260811 11:53:46.464627 29918 fs_manager.cc:359] Metadata directory not provided
I20260811 11:53:46.464723 29918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:53:46.470438 29918 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 11:53:46.473666 29941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:46.475456 29918 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:53:46.475651 29918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "9a64185803e64ad6b861206d82019fd3"
format_stamp: "Formatted at 2026-08-11 11:53:25 on dist-test-slave-bndk"
server_key: "b97048684df170465f45c4105c3e3925"
server_key_iv: "66b57707dccea05cbbee72fa6a93c3c3"
server_key_version: "encryptionkey@0"
I20260811 11:53:46.476339 29918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:53:46.490373 29918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:53:46.494546 29918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:53:46.494768 29918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:53:46.503417 29918 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:36175
I20260811 11:53:46.503778 30016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:36175 every 8 connection(s)
I20260811 11:53:46.505005 29918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:53:46.510685 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 29918
I20260811 11:53:46.511256 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:40733
--local_ip_for_outbound_sockets=127.3.221.65
--tserver_master_addrs=127.3.221.126:36175
--webserver_port=45867
--webserver_interface=127.3.221.65
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:53:46.511603 30018 sys_catalog.cc:263] Verifying existing consensus state
I20260811 11:53:46.514050 30018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3: Bootstrap starting.
I20260811 11:53:46.532845 30018 log.cc:824] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3: Log is configured to *not* fsync() on all Append() calls
I20260811 11:53:46.544152 30018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:53:46.544831 30018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3: Bootstrap complete.
I20260811 11:53:46.549275 30018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a64185803e64ad6b861206d82019fd3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36175 } }
I20260811 11:53:46.549751 30018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a64185803e64ad6b861206d82019fd3, State: Initialized, Role: FOLLOWER
I20260811 11:53:46.550293 30018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a64185803e64ad6b861206d82019fd3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36175 } }
I20260811 11:53:46.550419 30018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:53:46.550495 30018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:53:46.550590 30018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 1 FOLLOWER]: Advancing to term 2
I20260811 11:53:46.552672 30018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a64185803e64ad6b861206d82019fd3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36175 } }
I20260811 11:53:46.553032 30018 leader_election.cc:302] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a64185803e64ad6b861206d82019fd3; no voters:
I20260811 11:53:46.553406 30018 leader_election.cc:288] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 11:53:46.553537 30022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 11:53:46.553736 30022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 2 LEADER]: Becoming Leader. State: Replica: 9a64185803e64ad6b861206d82019fd3, State: Running, Role: LEADER
I20260811 11:53:46.554122 30022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a64185803e64ad6b861206d82019fd3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36175 } }
I20260811 11:53:46.554760 30018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:53:46.555545 30024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9a64185803e64ad6b861206d82019fd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a64185803e64ad6b861206d82019fd3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36175 } } }
I20260811 11:53:46.555680 30024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [sys.catalog]: This master's current role is: LEADER
I20260811 11:53:46.556391 30025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a64185803e64ad6b861206d82019fd3. Latest consensus state: current_term: 2 leader_uuid: "9a64185803e64ad6b861206d82019fd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a64185803e64ad6b861206d82019fd3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36175 } } }
I20260811 11:53:46.556497 30025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [sys.catalog]: This master's current role is: LEADER
I20260811 11:53:46.556448 30033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:53:46.562400 30033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:53:46.563133 30033 catalog_manager.cc:1259] Loaded cluster ID: dd85785dea4a43dfbe9da43ed0c7f4dd
I20260811 11:53:46.563202 30033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:53:46.565961 30033 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:53:46.566782 30033 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3: Loaded TSK: 0
I20260811 11:53:46.590777 30033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3: Generated new TSK 1
I20260811 11:53:46.591118 30033 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 11:53:46.649875 30020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:53:46.650187 30020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:53:46.650238 30020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:53:46.655119 30020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:53:46.655252 30020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:53:46.655341 30020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:53:46.662536 30020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:40733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=45867
--tserver_master_addrs=127.3.221.126:36175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:53:46.663666 30020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:53:46.664714 30020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:53:46.674829 30060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:46.675029 30020 server_base.cc:1034] running on GCE node
W20260811 11:53:46.674840 30062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:46.674860 30059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:46.675747 30020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:53:46.676661 30020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:53:46.677892 30020 hybrid_clock.cc:648] HybridClock initialized: now 1786449226677865 us; error 36 us; skew 500 ppm
Aug 11 11:53:46 dist-test-slave-bndk krb5kdc[29033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449226, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:53:46.681782 30020 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:53:46.682830 30020 webserver.cc:460] Webserver started at http://127.3.221.65:45867/ using document root <none> and password file <none>
I20260811 11:53:46.683444 30020 fs_manager.cc:359] Metadata directory not provided
I20260811 11:53:46.683525 30020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:53:46.689488 30020 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 11:53:46.693183 30073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:46.695504 30020 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 11:53:46.695677 30020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "87a977b6ceef49e988b94b6d5c3215fe"
format_stamp: "Formatted at 2026-08-11 11:53:25 on dist-test-slave-bndk"
server_key: "526e1c04fb107212ed2f7922088fdc89"
server_key_iv: "c5f6d42994686adb4bd9d8bace33227d"
server_key_version: "encryptionkey@0"
I20260811 11:53:46.696547 30020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:53:46.730877 30020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:53:46.736212 30020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:53:46.736481 30020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:53:46.737396 30020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:53:46.739102 30085 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260811 11:53:46.743821 30020 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 11:53:46.743927 30020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260811 11:53:46.744014 30020 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 11:53:46.746001 30020 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 11:53:46.746062 30020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260811 11:53:46.746444 30085 tablet_bootstrap.cc:492] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe: Bootstrap starting.
I20260811 11:53:46.756628 30020 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:40733
I20260811 11:53:46.756742 30274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:40733 every 8 connection(s)
I20260811 11:53:46.757725 30020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:53:46.765202 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 30020
I20260811 11:53:46.765699 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:38893
--local_ip_for_outbound_sockets=127.3.221.66
--tserver_master_addrs=127.3.221.126:36175
--webserver_port=36053
--webserver_interface=127.3.221.66
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:53:46.768514 30085 log.cc:824] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe: Log is configured to *not* fsync() on all Append() calls
Aug 11 11:53:46 dist-test-slave-bndk krb5kdc[29033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449226, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:53:46.780941 30085 tablet_bootstrap.cc:492] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:53:46.781481 30085 tablet_bootstrap.cc:492] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe: Bootstrap complete.
I20260811 11:53:46.782094 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:46.760494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.65:54257)
0811 11:53:46.761161 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:46.761259 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:46.762151 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:46.763724 (+ 1573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:46.763731 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:46.764144 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:53:46.764664 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:46.764681 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:46.767628 (+ 2947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:46.767633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:46.769547 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:46.769571 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:46.770655 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:46.770675 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:46.770919 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:46.771688 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:53:46.771710 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:53:46.774330 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:53:46.777778 (+ 3448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:46.777787 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:46.777802 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:46.778210 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:46.778689 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:46.778692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:46.778694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:46.778871 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:46.779729 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:53:46.779736 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:53:46.780056 (+ 320us) client_negotiation.cc:769] Sending connection context
0811 11:53:46.780324 (+ 268us) client_negotiation.cc:240] Negotiation successful
0811 11:53:46.780829 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":142,"threads_started":1}
I20260811 11:53:46.782096 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:46.760479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:54257 (local address 127.3.221.126:36175)
0811 11:53:46.760966 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:46.760988 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:46.761402 (+ 414us) server_negotiation.cc:407] Connection header received
0811 11:53:46.762628 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:46.762643 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:46.763101 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:46.763587 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:46.764861 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:46.767263 (+ 2402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:46.769790 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:46.770349 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:46.770508 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:46.774766 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:46.774778 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:46.774807 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:46.774861 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:46.777595 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:46.778378 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:46.778386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:46.778393 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:46.778575 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:46.778999 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:46.779003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:46.779005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:46.779587 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:46.779759 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:46.780495 (+ 736us) server_negotiation.cc:299] Negotiation successful
0811 11:53:46.780845 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":157,"threads_started":1}
I20260811 11:53:46.782878 30085 ts_tablet_manager.cc:1397] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe: Time spent bootstrapping tablet: real 0.037s user 0.022s sys 0.011s
I20260811 11:53:46.785794 30275 heartbeater.cc:347] Connected to a master server at 127.3.221.126:36175
I20260811 11:53:46.786103 30275 heartbeater.cc:464] Registering TS with master...
I20260811 11:53:46.786855 30275 heartbeater.cc:511] Master 127.3.221.126:36175 requested a full tablet report, sending...
I20260811 11:53:46.786572 30085 raft_consensus.cc:348] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } }
I20260811 11:53:46.787257 30085 raft_consensus.cc:729] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87a977b6ceef49e988b94b6d5c3215fe, State: Initialized, Role: FOLLOWER
I20260811 11:53:46.787911 30085 consensus_queue.cc:260] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } }
I20260811 11:53:46.789335 30085 ts_tablet_manager.cc:1428] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260811 11:53:46.792448 29965 ts_manager.cc:194] Registered new tserver with Master: 87a977b6ceef49e988b94b6d5c3215fe (127.3.221.65:40733)
I20260811 11:53:46.796882 29965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:54257
I20260811 11:53:46.799788 30275 heartbeater.cc:503] Master 127.3.221.126:36175 was elected leader, sending a full tablet report...
Aug 11 11:53:46 dist-test-slave-bndk krb5kdc[29033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449226, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for kudu/127.3.221.65@KRBTEST.COM
I20260811 11:53:46.808652 30285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:46.796324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:40733 (local address 127.0.0.1:48520)
0811 11:53:46.796784 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:46.796834 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:46.796982 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:46.797752 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:46.797758 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:46.798199 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:53:46.798341 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:46.798350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:46.799757 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:46.799760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:46.800617 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:46.800627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:46.801263 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:46.801267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:46.801389 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:46.802007 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:53:46.802023 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:53:46.803740 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:53:46.806616 (+ 2876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:46.806637 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:46.806643 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:46.806926 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:46.807324 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:46.807327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:46.807328 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:46.807432 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:46.808016 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:53:46.808022 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:53:46.808149 (+ 127us) client_negotiation.cc:769] Sending connection context
0811 11:53:46.808298 (+ 149us) client_negotiation.cc:240] Negotiation successful
0811 11:53:46.808502 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":143,"threads_started":1}
I20260811 11:53:46.808794 30286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:46.796446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48520 (local address 127.3.221.65:40733)
0811 11:53:46.797317 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:46.797327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:46.797350 (+ 23us) server_negotiation.cc:407] Connection header received
0811 11:53:46.797419 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:46.797423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:46.797591 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:46.797717 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:46.798476 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:46.799597 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:46.800732 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:46.801092 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:46.801191 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:46.803925 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:46.803936 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:46.803951 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:46.803980 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:46.806455 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:46.807047 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:46.807054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:46.807056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:46.807163 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:46.807526 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:46.807530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:46.807532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:46.807894 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:46.808031 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:46.808406 (+ 375us) server_negotiation.cc:299] Negotiation successful
0811 11:53:46.808603 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":118,"threads_started":1}
I20260811 11:53:46.810858 30166 tablet_service.cc:1514] Processing DeleteTablet for tablet 44ab24778d7140dda7a8e96f4effea3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:53:26 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:48520
I20260811 11:53:46.812465 30288 tablet_replica.cc:330] stopping tablet replica
I20260811 11:53:46.812991 30288 raft_consensus.cc:2229] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:53:46.813385 30288 raft_consensus.cc:2258] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:53:46.814761 30288 ts_tablet_manager.cc:1905] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:53:46.818988 30288 ts_tablet_manager.cc:1918] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:53:46.819103 30288 log.cc:1196] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/wal/wals/44ab24778d7140dda7a8e96f4effea3d
I20260811 11:53:46.819379 30288 ts_tablet_manager.cc:1939] T 44ab24778d7140dda7a8e96f4effea3d P 87a977b6ceef49e988b94b6d5c3215fe: Deleting consensus metadata
I20260811 11:53:46.821393 29942 catalog_manager.cc:4841] TS 87a977b6ceef49e988b94b6d5c3215fe (127.3.221.65:40733): tablet 44ab24778d7140dda7a8e96f4effea3d (table test-table [id=e6bacb02d47c40b2ab2ee8e95b420708]) successfully deleted
W20260811 11:53:46.912899 30279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:53:46.913283 30279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:53:46.913365 30279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:53:46.917783 30279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:53:46.917878 30279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:53:46.917964 30279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:53:46.923827 30279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:38893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=36053
--tserver_master_addrs=127.3.221.126:36175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:53:46.925137 30279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:53:46.926302 30279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:53:46.937625 30299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:46.937634 30298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:46.937736 30301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:46.937978 30279 server_base.cc:1034] running on GCE node
I20260811 11:53:46.938670 30279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:53:46.939409 30279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:53:46.940613 30279 hybrid_clock.cc:648] HybridClock initialized: now 1786449226940599 us; error 35 us; skew 500 ppm
Aug 11 11:53:46 dist-test-slave-bndk krb5kdc[29033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449226, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:53:46.943830 30279 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:53:46.944940 30279 webserver.cc:460] Webserver started at http://127.3.221.66:36053/ using document root <none> and password file <none>
I20260811 11:53:46.945783 30279 fs_manager.cc:359] Metadata directory not provided
I20260811 11:53:46.945889 30279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:53:46.951788 30279 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 11:53:46.955090 30315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:46.956874 30279 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:53:46.957041 30279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "0642a3ad98fe46869269ca187bd2d5fd"
format_stamp: "Formatted at 2026-08-11 11:53:25 on dist-test-slave-bndk"
server_key: "d9695daf9862f39a7446a81ef5768419"
server_key_iv: "b77e0c71a86320dba523b9ae71d57ca0"
server_key_version: "encryptionkey@0"
I20260811 11:53:46.957737 30279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:53:46.987443 30279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:53:46.992210 30279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:53:46.992651 30279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:53:46.993796 30279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:53:46.995193 30279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:53:46.995288 30279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:46.995364 30279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:53:46.995394 30279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:47.005414 30279 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:38893
I20260811 11:53:47.005623 30495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:38893 every 8 connection(s)
I20260811 11:53:47.007150 30279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:53:47.016942 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 30279
I20260811 11:53:47.017454 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:46597
--local_ip_for_outbound_sockets=127.3.221.67
--tserver_master_addrs=127.3.221.126:36175
--webserver_port=39321
--webserver_interface=127.3.221.67
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 11:53:47 dist-test-slave-bndk krb5kdc[29033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449226, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:53:47.027194 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:47.010132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:33773 (local address 127.3.221.126:36175)
0811 11:53:47.010376 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:47.010381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:47.011414 (+ 1033us) server_negotiation.cc:407] Connection header received
0811 11:53:47.013105 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:47.013108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:47.013182 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:47.013299 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:47.015349 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.015946 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.017324 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.017633 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.017733 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:47.021337 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:47.021341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:47.021343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:47.021383 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:47.023060 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:47.023945 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:47.023948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:47.023950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:47.024122 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:47.025522 (+ 1400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:47.025527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:47.025529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:47.025840 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:47.025979 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:47.026873 (+ 894us) server_negotiation.cc:299] Negotiation successful
0811 11:53:47.027049 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260811 11:53:47.028462 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:47.010533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.66:33773)
0811 11:53:47.011175 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:47.011247 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:47.012819 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:47.013726 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:47.013739 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:47.014326 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:53:47.015027 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:47.015046 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.016108 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.016112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:47.017164 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:47.017179 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.017809 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.017814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:47.018049 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:47.018766 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:53:47.018792 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:53:47.020812 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:53:47.023238 (+ 2426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:47.023249 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:47.023273 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:47.023789 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:47.024243 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:47.024246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:47.024248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:47.025307 (+ 1059us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:47.026006 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:53:47.026016 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:53:47.026482 (+ 466us) client_negotiation.cc:769] Sending connection context
0811 11:53:47.026759 (+ 277us) client_negotiation.cc:240] Negotiation successful
0811 11:53:47.027202 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":152,"threads_started":1}
I20260811 11:53:47.029978 30496 heartbeater.cc:347] Connected to a master server at 127.3.221.126:36175
I20260811 11:53:47.030292 30496 heartbeater.cc:464] Registering TS with master...
I20260811 11:53:47.031028 30496 heartbeater.cc:511] Master 127.3.221.126:36175 requested a full tablet report, sending...
I20260811 11:53:47.032589 29965 ts_manager.cc:194] Registered new tserver with Master: 0642a3ad98fe46869269ca187bd2d5fd (127.3.221.66:38893)
I20260811 11:53:47.033300 29965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:33773
W20260811 11:53:47.166967 30500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:53:47.167292 30500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:53:47.167361 30500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:53:47.172915 30500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:53:47.173032 30500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:53:47.173126 30500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:53:47.179998 30500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:46597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=39321
--tserver_master_addrs=127.3.221.126:36175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:53:47.181285 30500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:53:47.182404 30500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:53:47.195030 30508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:53:47.194926 30506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:47.194953 30500 server_base.cc:1034] running on GCE node
W20260811 11:53:47.194932 30512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:53:47.196004 30500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:53:47.196949 30500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:53:47.198163 30500 hybrid_clock.cc:648] HybridClock initialized: now 1786449227198130 us; error 51 us; skew 500 ppm
Aug 11 11:53:47 dist-test-slave-bndk krb5kdc[29033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449227, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:53:47.201547 30500 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:53:47.202643 30500 webserver.cc:460] Webserver started at http://127.3.221.67:39321/ using document root <none> and password file <none>
I20260811 11:53:47.203310 30500 fs_manager.cc:359] Metadata directory not provided
I20260811 11:53:47.203408 30500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:53:47.209395 30500 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 11:53:47.211902 30521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:47.213495 30500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:53:47.213634 30500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "dc6745c5de9b426690d9e3ec5ae80bb9"
format_stamp: "Formatted at 2026-08-11 11:53:25 on dist-test-slave-bndk"
server_key: "23dbd358c7ce80ac09e529116902bb89"
server_key_iv: "c8b93a517fd4bbcdf50f1a4c2ff8e07f"
server_key_version: "encryptionkey@0"
I20260811 11:53:47.214272 30500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:53:47.234822 30500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:53:47.239099 30500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:53:47.239319 30500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:53:47.240015 30500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:53:47.240970 30500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:53:47.241021 30500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:47.241079 30500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:53:47.241107 30500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:53:47.253954 30500 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:46597
I20260811 11:53:47.253979 30693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:46597 every 8 connection(s)
I20260811 11:53:47.255543 30500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:53:47.257808 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 30500
Aug 11 11:53:47 dist-test-slave-bndk krb5kdc[29033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449227, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:53:47.267516 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:47.258911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54212 (local address 127.3.221.126:36175)
0811 11:53:47.259232 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:47.259236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:47.259254 (+ 18us) server_negotiation.cc:407] Connection header received
0811 11:53:47.259314 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:47.259317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:47.259366 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:47.259463 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:47.260243 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.261020 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.261997 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.262348 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.262437 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:47.264079 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:47.264082 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:47.264085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:47.264114 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:47.265814 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:47.266236 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:47.266240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:47.266241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:47.266320 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:47.266578 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:47.266581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:47.266583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:47.266808 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:47.267022 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:47.267184 (+ 162us) server_negotiation.cc:299] Negotiation successful
0811 11:53:47.267314 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260811 11:53:47.271847 30700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:47.259019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:47337 (local address 127.3.221.126:36175)
0811 11:53:47.259262 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:47.259266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:47.259284 (+ 18us) server_negotiation.cc:407] Connection header received
0811 11:53:47.260615 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:47.260618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:47.260662 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:47.260745 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:47.262388 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.263098 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.264479 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.264718 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.264804 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:47.267360 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:47.267363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:47.267364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:47.267387 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:47.268970 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:47.269747 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:47.269751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:47.269752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:47.269855 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:47.270446 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:47.270453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:47.270455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:47.270793 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:47.270899 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:47.271556 (+ 657us) server_negotiation.cc:299] Negotiation successful
0811 11:53:47.271692 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":94,"threads_started":1}
I20260811 11:53:47.272825 30698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:47.258253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.67:47337)
0811 11:53:47.258875 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:47.258937 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:47.260345 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:47.260984 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:47.260994 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:47.261489 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:53:47.262137 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:47.262157 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.263219 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.263221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:47.264351 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:47.264364 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.264875 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.264880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:47.265069 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:47.265586 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:53:47.265604 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:53:47.267178 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:53:47.269111 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:47.269118 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:47.269132 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:47.269536 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:47.270004 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:53:47.270007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:53:47.270008 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:53:47.270173 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:53:47.270935 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:53:47.270941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:53:47.271266 (+ 325us) client_negotiation.cc:769] Sending connection context
0811 11:53:47.271461 (+ 195us) client_negotiation.cc:240] Negotiation successful
0811 11:53:47.271765 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":144,"threads_started":1}
I20260811 11:53:47.274284 30694 heartbeater.cc:347] Connected to a master server at 127.3.221.126:36175
I20260811 11:53:47.274587 30694 heartbeater.cc:464] Registering TS with master...
I20260811 11:53:47.275182 30694 heartbeater.cc:511] Master 127.3.221.126:36175 requested a full tablet report, sending...
I20260811 11:53:47.276453 29965 ts_manager.cc:194] Registered new tserver with Master: dc6745c5de9b426690d9e3ec5ae80bb9 (127.3.221.67:46597)
I20260811 11:53:47.277035 29965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:47337
I20260811 11:53:47.282068 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:53:47.288290 30700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:47.282884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54216 (local address 127.3.221.126:36175)
0811 11:53:47.283147 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:47.283153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:47.283250 (+ 97us) server_negotiation.cc:407] Connection header received
0811 11:53:47.283432 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:47.283434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:47.283582 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:47.283681 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:53:47.283942 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.284749 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.286384 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.286658 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.286766 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:47.287221 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:53:47.287574 (+ 353us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0811 11:53:47.288092 (+ 518us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54216: token expired
Metrics: {"server-negotiator.queue_time_us":183}
W20260811 11:53:47.288424 30700 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54216: token expired
W20260811 11:53:47.289000 29855 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.221.126:36175)
I20260811 11:53:47.298686 30700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:47.289707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54220 (local address 127.3.221.126:36175)
0811 11:53:47.289842 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:47.289871 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:47.289887 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:53:47.289928 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:47.289930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:47.289982 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:47.290054 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:53:47.290769 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.291299 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.292369 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.292715 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.292856 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:47.295050 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:53:47.295054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:53:47.295056 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:53:47.295099 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:53:47.296617 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:47.297149 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:47.297152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:47.297154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:47.297313 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:53:47.297649 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:53:47.297652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:53:47.297653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:53:47.297884 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:53:47.298007 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:47.298383 (+ 376us) server_negotiation.cc:299] Negotiation successful
0811 11:53:47.298509 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 11:53:47.301992 29965 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:53:47.305675 29965 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:53:47.316177 30166 tablet_service.cc:1467] Processing CreateTablet for tablet 6141b6509ab64fdb83b200879d70da6e (DEFAULT_TABLE table=test-table [id=91530941630c4602b8b422503ada0898]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:53:47.316960 30166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6141b6509ab64fdb83b200879d70da6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:53:47.320705 30715 tablet_bootstrap.cc:492] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe: Bootstrap starting.
I20260811 11:53:47.321923 30715 tablet_bootstrap.cc:654] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe: Neither blocks nor log segments found. Creating new log.
I20260811 11:53:47.321902 30713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:47.315305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:46597 (local address 127.0.0.1:60212)
0811 11:53:47.315717 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:47.315730 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:47.315826 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:47.316572 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:47.316576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:47.316595 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:53:47.316715 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:47.316721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.318270 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.318274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:47.319653 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:47.319667 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.321348 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.321352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:47.321453 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:47.321508 (+ 55us) client_negotiation.cc:769] Sending connection context
0811 11:53:47.321593 (+ 85us) client_negotiation.cc:240] Negotiation successful
0811 11:53:47.321694 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":5,"thread_start_us":90,"threads_started":1}
I20260811 11:53:47.321966 30714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:47.315711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60212 (local address 127.3.221.67:46597)
0811 11:53:47.316051 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:47.316065 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:47.316090 (+ 25us) server_negotiation.cc:407] Connection header received
0811 11:53:47.316169 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:47.316174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:47.316394 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:47.316864 (+ 470us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:53:47.317030 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.318133 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.319841 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.321143 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.321340 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:47.321500 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:47.321689 (+ 189us) server_negotiation.cc:299] Negotiation successful
0811 11:53:47.321809 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":110,"threads_started":1}
I20260811 11:53:47.321915 30709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:47.315051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:38893 (local address 127.0.0.1:39818)
0811 11:53:47.315474 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:47.315491 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:47.315585 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:47.316722 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:47.316725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:47.316744 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:53:47.316835 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:47.316842 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.318100 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.318104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:47.319662 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:47.319693 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.321530 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.321534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:47.321622 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:47.321641 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 11:53:47.321699 (+ 58us) client_negotiation.cc:240] Negotiation successful
0811 11:53:47.321814 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":36,"thread_start_us":105,"threads_started":1}
I20260811 11:53:47.322660 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:47.315201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39818 (local address 127.3.221.66:38893)
0811 11:53:47.316064 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:47.316073 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:47.316092 (+ 19us) server_negotiation.cc:407] Connection header received
0811 11:53:47.316165 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:47.316170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:47.316355 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:47.316522 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:53:47.316955 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.317949 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.319959 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.321134 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.321281 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:47.321459 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:47.322395 (+ 936us) server_negotiation.cc:299] Negotiation successful
0811 11:53:47.322506 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":137,"threads_started":1}
I20260811 11:53:47.324038 30715 tablet_bootstrap.cc:492] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe: No bootstrap required, opened a new log
I20260811 11:53:47.324142 30715 ts_tablet_manager.cc:1397] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260811 11:53:47.324615 30715 raft_consensus.cc:348] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } }
I20260811 11:53:47.324738 30715 raft_consensus.cc:374] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:53:47.324774 30715 raft_consensus.cc:729] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87a977b6ceef49e988b94b6d5c3215fe, State: Initialized, Role: FOLLOWER
I20260811 11:53:47.324913 30715 consensus_queue.cc:260] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } }
I20260811 11:53:47.325302 30715 ts_tablet_manager.cc:1428] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 11:53:47.325637 30394 tablet_service.cc:1467] Processing CreateTablet for tablet 6141b6509ab64fdb83b200879d70da6e (DEFAULT_TABLE table=test-table [id=91530941630c4602b8b422503ada0898]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:53:47.325637 30596 tablet_service.cc:1467] Processing CreateTablet for tablet 6141b6509ab64fdb83b200879d70da6e (DEFAULT_TABLE table=test-table [id=91530941630c4602b8b422503ada0898]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:53:47.326728 30394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6141b6509ab64fdb83b200879d70da6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:53:47.326783 30596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6141b6509ab64fdb83b200879d70da6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:53:47.334689 30725 tablet_bootstrap.cc:492] T 6141b6509ab64fdb83b200879d70da6e P dc6745c5de9b426690d9e3ec5ae80bb9: Bootstrap starting.
I20260811 11:53:47.336534 30726 tablet_bootstrap.cc:492] T 6141b6509ab64fdb83b200879d70da6e P 0642a3ad98fe46869269ca187bd2d5fd: Bootstrap starting.
I20260811 11:53:47.337100 30725 tablet_bootstrap.cc:654] T 6141b6509ab64fdb83b200879d70da6e P dc6745c5de9b426690d9e3ec5ae80bb9: Neither blocks nor log segments found. Creating new log.
I20260811 11:53:47.338363 30725 log.cc:824] T 6141b6509ab64fdb83b200879d70da6e P dc6745c5de9b426690d9e3ec5ae80bb9: Log is configured to *not* fsync() on all Append() calls
I20260811 11:53:47.339313 30726 tablet_bootstrap.cc:654] T 6141b6509ab64fdb83b200879d70da6e P 0642a3ad98fe46869269ca187bd2d5fd: Neither blocks nor log segments found. Creating new log.
I20260811 11:53:47.340188 30726 log.cc:824] T 6141b6509ab64fdb83b200879d70da6e P 0642a3ad98fe46869269ca187bd2d5fd: Log is configured to *not* fsync() on all Append() calls
I20260811 11:53:47.340611 30725 tablet_bootstrap.cc:492] T 6141b6509ab64fdb83b200879d70da6e P dc6745c5de9b426690d9e3ec5ae80bb9: No bootstrap required, opened a new log
I20260811 11:53:47.340862 30725 ts_tablet_manager.cc:1397] T 6141b6509ab64fdb83b200879d70da6e P dc6745c5de9b426690d9e3ec5ae80bb9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 11:53:47.342310 30726 tablet_bootstrap.cc:492] T 6141b6509ab64fdb83b200879d70da6e P 0642a3ad98fe46869269ca187bd2d5fd: No bootstrap required, opened a new log
I20260811 11:53:47.342527 30726 ts_tablet_manager.cc:1397] T 6141b6509ab64fdb83b200879d70da6e P 0642a3ad98fe46869269ca187bd2d5fd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 11:53:47.344152 30725 raft_consensus.cc:348] T 6141b6509ab64fdb83b200879d70da6e P dc6745c5de9b426690d9e3ec5ae80bb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } }
I20260811 11:53:47.344421 30725 raft_consensus.cc:374] T 6141b6509ab64fdb83b200879d70da6e P dc6745c5de9b426690d9e3ec5ae80bb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:53:47.344475 30725 raft_consensus.cc:729] T 6141b6509ab64fdb83b200879d70da6e P dc6745c5de9b426690d9e3ec5ae80bb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc6745c5de9b426690d9e3ec5ae80bb9, State: Initialized, Role: FOLLOWER
I20260811 11:53:47.344960 30725 consensus_queue.cc:260] T 6141b6509ab64fdb83b200879d70da6e P dc6745c5de9b426690d9e3ec5ae80bb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } }
I20260811 11:53:47.345778 30694 heartbeater.cc:503] Master 127.3.221.126:36175 was elected leader, sending a full tablet report...
I20260811 11:53:47.346097 30725 ts_tablet_manager.cc:1428] T 6141b6509ab64fdb83b200879d70da6e P dc6745c5de9b426690d9e3ec5ae80bb9: Time spent starting tablet: real 0.005s user 0.003s sys 0.004s
I20260811 11:53:47.345925 30726 raft_consensus.cc:348] T 6141b6509ab64fdb83b200879d70da6e P 0642a3ad98fe46869269ca187bd2d5fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } }
I20260811 11:53:47.346177 30726 raft_consensus.cc:374] T 6141b6509ab64fdb83b200879d70da6e P 0642a3ad98fe46869269ca187bd2d5fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:53:47.346247 30726 raft_consensus.cc:729] T 6141b6509ab64fdb83b200879d70da6e P 0642a3ad98fe46869269ca187bd2d5fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0642a3ad98fe46869269ca187bd2d5fd, State: Initialized, Role: FOLLOWER
I20260811 11:53:47.346690 30726 consensus_queue.cc:260] T 6141b6509ab64fdb83b200879d70da6e P 0642a3ad98fe46869269ca187bd2d5fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } }
I20260811 11:53:47.347713 30496 heartbeater.cc:503] Master 127.3.221.126:36175 was elected leader, sending a full tablet report...
I20260811 11:53:47.348027 30726 ts_tablet_manager.cc:1428] T 6141b6509ab64fdb83b200879d70da6e P 0642a3ad98fe46869269ca187bd2d5fd: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260811 11:53:47.430164 30724 raft_consensus.cc:482] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:53:47.430429 30724 raft_consensus.cc:504] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } }
I20260811 11:53:47.431815 30724 leader_election.cc:288] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0642a3ad98fe46869269ca187bd2d5fd (127.3.221.66:38893), dc6745c5de9b426690d9e3ec5ae80bb9 (127.3.221.67:46597)
I20260811 11:53:47.435644 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:47.432060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:58817 (local address 127.3.221.66:38893)
0811 11:53:47.432195 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:47.432201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:47.432438 (+ 237us) server_negotiation.cc:407] Connection header received
0811 11:53:47.432630 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:47.432634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:47.432693 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:47.432798 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:53:47.433082 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.433629 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.434679 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.435082 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.435192 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:47.435252 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:47.435448 (+ 196us) server_negotiation.cc:299] Negotiation successful
0811 11:53:47.435520 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 11:53:47.435647 30735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:47.431905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:38893 (local address 127.3.221.65:58817)
0811 11:53:47.432299 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:47.432312 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:47.432477 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:47.432822 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:47.432826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:47.432845 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:53:47.432970 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:47.432976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.433738 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.433741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:47.434551 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:47.434561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.435200 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.435203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:47.435309 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:47.435347 (+ 38us) client_negotiation.cc:769] Sending connection context
0811 11:53:47.435416 (+ 69us) client_negotiation.cc:240] Negotiation successful
0811 11:53:47.435492 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":116,"threads_started":1}
I20260811 11:53:47.435813 30714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:47.432502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:33279 (local address 127.3.221.67:46597)
0811 11:53:47.432641 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:47.432644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:47.432829 (+ 185us) server_negotiation.cc:407] Connection header received
0811 11:53:47.432921 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:47.432924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:47.432966 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:47.433028 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:53:47.433258 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.433826 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.434929 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.435324 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.435436 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:47.435486 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:47.435635 (+ 149us) server_negotiation.cc:299] Negotiation successful
0811 11:53:47.435703 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 11:53:47.435951 30736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:47.432277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:46597 (local address 127.3.221.65:33279)
0811 11:53:47.432738 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:47.432750 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:53:47.432836 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:53:47.433053 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:53:47.433056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:53:47.433071 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:53:47.433157 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:47.433162 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.433972 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.433975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:47.434812 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:53:47.434822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.435436 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.435440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:53:47.435526 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:47.435547 (+ 21us) client_negotiation.cc:769] Sending connection context
0811 11:53:47.435612 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 11:53:47.435680 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":211,"threads_started":1}
I20260811 11:53:47.436503 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6141b6509ab64fdb83b200879d70da6e" candidate_uuid: "87a977b6ceef49e988b94b6d5c3215fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0642a3ad98fe46869269ca187bd2d5fd" is_pre_election: true
I20260811 11:53:47.436522 30632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6141b6509ab64fdb83b200879d70da6e" candidate_uuid: "87a977b6ceef49e988b94b6d5c3215fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" is_pre_election: true
I20260811 11:53:47.436810 30422 raft_consensus.cc:2454] T 6141b6509ab64fdb83b200879d70da6e P 0642a3ad98fe46869269ca187bd2d5fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87a977b6ceef49e988b94b6d5c3215fe in term 0.
I20260811 11:53:47.436811 30632 raft_consensus.cc:2454] T 6141b6509ab64fdb83b200879d70da6e P dc6745c5de9b426690d9e3ec5ae80bb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87a977b6ceef49e988b94b6d5c3215fe in term 0.
I20260811 11:53:47.437330 30079 leader_election.cc:302] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87a977b6ceef49e988b94b6d5c3215fe, dc6745c5de9b426690d9e3ec5ae80bb9; no voters:
I20260811 11:53:47.437623 30724 raft_consensus.cc:2781] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:53:47.437714 30724 raft_consensus.cc:482] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:53:47.437765 30724 raft_consensus.cc:3037] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:53:47.438799 30724 raft_consensus.cc:504] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } }
I20260811 11:53:47.439141 30724 leader_election.cc:288] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [CANDIDATE]: Term 1 election: Requested vote from peers 0642a3ad98fe46869269ca187bd2d5fd (127.3.221.66:38893), dc6745c5de9b426690d9e3ec5ae80bb9 (127.3.221.67:46597)
I20260811 11:53:47.439594 30632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6141b6509ab64fdb83b200879d70da6e" candidate_uuid: "87a977b6ceef49e988b94b6d5c3215fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9"
I20260811 11:53:47.439728 30632 raft_consensus.cc:3037] T 6141b6509ab64fdb83b200879d70da6e P dc6745c5de9b426690d9e3ec5ae80bb9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:53:47.439908 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6141b6509ab64fdb83b200879d70da6e" candidate_uuid: "87a977b6ceef49e988b94b6d5c3215fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0642a3ad98fe46869269ca187bd2d5fd"
I20260811 11:53:47.440045 30422 raft_consensus.cc:3037] T 6141b6509ab64fdb83b200879d70da6e P 0642a3ad98fe46869269ca187bd2d5fd [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:53:47.441144 30632 raft_consensus.cc:2454] T 6141b6509ab64fdb83b200879d70da6e P dc6745c5de9b426690d9e3ec5ae80bb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87a977b6ceef49e988b94b6d5c3215fe in term 1.
I20260811 11:53:47.441481 30422 raft_consensus.cc:2454] T 6141b6509ab64fdb83b200879d70da6e P 0642a3ad98fe46869269ca187bd2d5fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87a977b6ceef49e988b94b6d5c3215fe in term 1.
I20260811 11:53:47.441540 30079 leader_election.cc:302] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87a977b6ceef49e988b94b6d5c3215fe, dc6745c5de9b426690d9e3ec5ae80bb9; no voters:
I20260811 11:53:47.441725 30724 raft_consensus.cc:2781] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:53:47.442047 30724 raft_consensus.cc:686] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 1 LEADER]: Becoming Leader. State: Replica: 87a977b6ceef49e988b94b6d5c3215fe, State: Running, Role: LEADER
I20260811 11:53:47.442340 30724 consensus_queue.cc:237] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } }
I20260811 11:53:47.445699 29965 catalog_manager.cc:5495] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 87a977b6ceef49e988b94b6d5c3215fe (127.3.221.65). New cstate: current_term: 1 leader_uuid: "87a977b6ceef49e988b94b6d5c3215fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } health_report { overall_health: HEALTHY } } }
I20260811 11:53:47.482312 30742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:47.477894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48526 (local address 127.3.221.65:40733)
0811 11:53:47.478255 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0811 11:53:47.478264 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 11:53:47.478279 (+ 15us) server_negotiation.cc:407] Connection header received
0811 11:53:47.478373 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:53:47.478375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:53:47.478433 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:53:47.478546 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:53:47.478877 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.479412 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.480319 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:53:47.480566 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:53:47.480659 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:53:47.480992 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:53:47.481574 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:53:47.481670 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 11:53:47.481830 (+ 160us) server_negotiation.cc:299] Negotiation successful
0811 11:53:47.481901 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":163,"threads_started":1}
I20260811 11:53:47.487984 30422 raft_consensus.cc:1261] T 6141b6509ab64fdb83b200879d70da6e P 0642a3ad98fe46869269ca187bd2d5fd [term 1 FOLLOWER]: Refusing update from remote peer 87a977b6ceef49e988b94b6d5c3215fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:53:47.488015 30632 raft_consensus.cc:1261] T 6141b6509ab64fdb83b200879d70da6e P dc6745c5de9b426690d9e3ec5ae80bb9 [term 1 FOLLOWER]: Refusing update from remote peer 87a977b6ceef49e988b94b6d5c3215fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:53:47.489082 30738 consensus_queue.cc:1035] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:53:47.489416 30724 consensus_queue.cc:1035] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:53:47.500548 30745 mvcc.cc:205] Tried to move back new op lower bound from 7317296035781201920 to 7317296035606654976. Current Snapshot: MvccSnapshot[applied={T|T < 7317296035781201920}]
I20260811 11:53:47.510700 29965 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:53:47.510926 29965 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:53:47.514091 29965 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3: Sending DeleteTablet for 3 replicas of tablet 6141b6509ab64fdb83b200879d70da6e
I20260811 11:53:47.515026 30394 tablet_service.cc:1514] Processing DeleteTablet for tablet 6141b6509ab64fdb83b200879d70da6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:53:47 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:39818
I20260811 11:53:47.515408 30596 tablet_service.cc:1514] Processing DeleteTablet for tablet 6141b6509ab64fdb83b200879d70da6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:53:47 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:60212
I20260811 11:53:47.515865 30166 tablet_service.cc:1514] Processing DeleteTablet for tablet 6141b6509ab64fdb83b200879d70da6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:53:47 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:48520
I20260811 11:53:47.516028 30756 tablet_replica.cc:330] T 6141b6509ab64fdb83b200879d70da6e P dc6745c5de9b426690d9e3ec5ae80bb9: stopping tablet replica
I20260811 11:53:47.516456 30756 raft_consensus.cc:2229] T 6141b6509ab64fdb83b200879d70da6e P dc6745c5de9b426690d9e3ec5ae80bb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:53:47.516616 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 30020
I20260811 11:53:47.516999 30756 raft_consensus.cc:2258] T 6141b6509ab64fdb83b200879d70da6e P dc6745c5de9b426690d9e3ec5ae80bb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:53:47.527209 30756 ts_tablet_manager.cc:1905] T 6141b6509ab64fdb83b200879d70da6e P dc6745c5de9b426690d9e3ec5ae80bb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:53:47.531066 30756 ts_tablet_manager.cc:1918] T 6141b6509ab64fdb83b200879d70da6e P dc6745c5de9b426690d9e3ec5ae80bb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:53:47.531193 30756 log.cc:1196] T 6141b6509ab64fdb83b200879d70da6e P dc6745c5de9b426690d9e3ec5ae80bb9: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/6141b6509ab64fdb83b200879d70da6e
I20260811 11:53:47.531522 30756 ts_tablet_manager.cc:1939] T 6141b6509ab64fdb83b200879d70da6e P dc6745c5de9b426690d9e3ec5ae80bb9: Deleting consensus metadata
I20260811 11:53:47.533506 30755 tablet_replica.cc:330] T 6141b6509ab64fdb83b200879d70da6e P 0642a3ad98fe46869269ca187bd2d5fd: stopping tablet replica
I20260811 11:53:47.533627 29948 catalog_manager.cc:4841] TS dc6745c5de9b426690d9e3ec5ae80bb9 (127.3.221.67:46597): tablet 6141b6509ab64fdb83b200879d70da6e (table test-table [id=91530941630c4602b8b422503ada0898]) successfully deleted
I20260811 11:53:47.533918 30755 raft_consensus.cc:2229] T 6141b6509ab64fdb83b200879d70da6e P 0642a3ad98fe46869269ca187bd2d5fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:53:47.534265 30755 raft_consensus.cc:2258] T 6141b6509ab64fdb83b200879d70da6e P 0642a3ad98fe46869269ca187bd2d5fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:53:47.535905 30755 ts_tablet_manager.cc:1905] T 6141b6509ab64fdb83b200879d70da6e P 0642a3ad98fe46869269ca187bd2d5fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:53:47.538298 30755 ts_tablet_manager.cc:1918] T 6141b6509ab64fdb83b200879d70da6e P 0642a3ad98fe46869269ca187bd2d5fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:53:47.538403 30755 log.cc:1196] T 6141b6509ab64fdb83b200879d70da6e P 0642a3ad98fe46869269ca187bd2d5fd: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/6141b6509ab64fdb83b200879d70da6e
I20260811 11:53:47.538697 30755 ts_tablet_manager.cc:1939] T 6141b6509ab64fdb83b200879d70da6e P 0642a3ad98fe46869269ca187bd2d5fd: Deleting consensus metadata
I20260811 11:53:47.539934 29945 catalog_manager.cc:4841] TS 0642a3ad98fe46869269ca187bd2d5fd (127.3.221.66:38893): tablet 6141b6509ab64fdb83b200879d70da6e (table test-table [id=91530941630c4602b8b422503ada0898]) successfully deleted
W20260811 11:53:47.540099 29942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.221.65:40733 (error 108)
I20260811 11:53:47.540249 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 30279
I20260811 11:53:47.556099 30709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:53:47.554407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:40733 (local address 127.0.0.1:48532)
0811 11:53:47.555874 (+ 1467us) negotiation.cc:107] Waiting for socket to connect
0811 11:53:47.555982 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.65:40733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1350}
W20260811 11:53:47.556332 29942 catalog_manager.cc:4584] TS 87a977b6ceef49e988b94b6d5c3215fe (127.3.221.65:40733): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6141b6509ab64fdb83b200879d70da6e: Network error: Client connection negotiation failed: client connection to 127.3.221.65:40733: connect: Connection refused (error 111)
I20260811 11:53:47.562436 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 30500
I20260811 11:53:47.592211 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 29918
I20260811 11:54:07.617590 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:36175
--webserver_interface=127.3.221.126
--webserver_port=36357
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:36175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:07.750825 30763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:07.751169 30763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:07.751232 30763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:07.756170 30763 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 11:54:07.756263 30763 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 11:54:07.756279 30763 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 11:54:07.756292 30763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:54:07.756311 30763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:07.756328 30763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:07.756350 30763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:54:07.756366 30763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:54:07.762336 30763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:36175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:36175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=36357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:07.763736 30763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:07.764878 30763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:07.774601 30771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:07.774590 30770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:07.774647 30763 server_base.cc:1034] running on GCE node
W20260811 11:54:07.774590 30773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:07.775460 30763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:07.776674 30763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:07.777865 30763 hybrid_clock.cc:648] HybridClock initialized: now 1786449247777849 us; error 32 us; skew 500 ppm
Aug 11 11:54:07 dist-test-slave-bndk krb5kdc[29033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449247, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:07.781104 30763 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:54:07.782163 30763 webserver.cc:460] Webserver started at http://127.3.221.126:36357/ using document root <none> and password file <none>
I20260811 11:54:07.782756 30763 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:07.782833 30763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:07.787429 30763 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 11:54:07.790033 30783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:07.791505 30763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:54:07.791636 30763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "9a64185803e64ad6b861206d82019fd3"
format_stamp: "Formatted at 2026-08-11 11:53:25 on dist-test-slave-bndk"
server_key: "b97048684df170465f45c4105c3e3925"
server_key_iv: "66b57707dccea05cbbee72fa6a93c3c3"
server_key_version: "encryptionkey@0"
I20260811 11:54:07.792292 30763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:07.834264 30763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:07.838714 30763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:07.838948 30763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:07.847004 30763 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:36175
I20260811 11:54:07.847350 30865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:36175 every 8 connection(s)
I20260811 11:54:07.848278 30763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:54:07.853176 30866 sys_catalog.cc:263] Verifying existing consensus state
I20260811 11:54:07.855121 30866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3: Bootstrap starting.
I20260811 11:54:07.858498 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 30763
I20260811 11:54:07.858974 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:40733
--local_ip_for_outbound_sockets=127.3.221.65
--tserver_master_addrs=127.3.221.126:36175
--webserver_port=45867
--webserver_interface=127.3.221.65
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:54:07.875252 30866 log.cc:824] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:07.887610 30866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:54:07.888039 30866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3: Bootstrap complete.
I20260811 11:54:07.893654 30866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a64185803e64ad6b861206d82019fd3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36175 } }
I20260811 11:54:07.894145 30866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a64185803e64ad6b861206d82019fd3, State: Initialized, Role: FOLLOWER
I20260811 11:54:07.894699 30866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a64185803e64ad6b861206d82019fd3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36175 } }
I20260811 11:54:07.894842 30866 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:54:07.894917 30866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:54:07.895017 30866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 2 FOLLOWER]: Advancing to term 3
I20260811 11:54:07.897293 30866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a64185803e64ad6b861206d82019fd3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36175 } }
I20260811 11:54:07.897652 30866 leader_election.cc:302] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a64185803e64ad6b861206d82019fd3; no voters:
I20260811 11:54:07.897992 30866 leader_election.cc:288] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260811 11:54:07.898110 30876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 3 FOLLOWER]: Leader election won for term 3
I20260811 11:54:07.898298 30876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [term 3 LEADER]: Becoming Leader. State: Replica: 9a64185803e64ad6b861206d82019fd3, State: Running, Role: LEADER
I20260811 11:54:07.898620 30876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a64185803e64ad6b861206d82019fd3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36175 } }
I20260811 11:54:07.899353 30866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:54:07.900213 30877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9a64185803e64ad6b861206d82019fd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a64185803e64ad6b861206d82019fd3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36175 } } }
I20260811 11:54:07.900359 30877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:07.900725 30888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:54:07.900754 30878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a64185803e64ad6b861206d82019fd3. Latest consensus state: current_term: 3 leader_uuid: "9a64185803e64ad6b861206d82019fd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a64185803e64ad6b861206d82019fd3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36175 } } }
I20260811 11:54:07.900949 30878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:07.905728 30888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:54:07.906412 30888 catalog_manager.cc:1259] Loaded cluster ID: dd85785dea4a43dfbe9da43ed0c7f4dd
I20260811 11:54:07.906477 30888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:54:07.908969 30888 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:54:07.909825 30888 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3: Loaded TSK: 0
I20260811 11:54:07.909921 30888 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9a64185803e64ad6b861206d82019fd3: Loaded TSK: 1
F20260811 11:54:07.910684 30888 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f40b952a9ad google::LogMessage::Fail() at ??:0
@ 0x7f40b952d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f40b952a4de google::LogMessage::Flush() at ??:0
@ 0x7f40b952ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f40bc1ee18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f40bc1e6f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f40c3e8f7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f40c3e6999c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f40c3e6a178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f40c3eabdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f40c3eca82b std::function<>::operator()() at ??:0
@ 0x7f40c3e69b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f40c3e6af3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f40c3e67579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f40c3eab640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f40bb77c3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f40bb77cc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f40bb77e3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f40bb76d512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f40bcdb16db start_thread at ??:0
@ 0x7f40b802861f clone at ??:0
W20260811 11:54:08.003882 30869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:08.004283 30869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:08.004352 30869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:08.010905 30869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:08.010987 30869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:08.011083 30869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:54:08.019944 30869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:40733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=45867
--tserver_master_addrs=127.3.221.126:36175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:08.021658 30869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:08.023147 30869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:08.037034 30954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:08.037318 30869 server_base.cc:1034] running on GCE node
W20260811 11:54:08.037025 30950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:08.037483 30957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:08.038102 30869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:08.039000 30869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:08.040179 30869 hybrid_clock.cc:648] HybridClock initialized: now 1786449248040149 us; error 46 us; skew 500 ppm
Aug 11 11:54:08 dist-test-slave-bndk krb5kdc[29033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:08.043354 30869 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:54:08.044351 30869 webserver.cc:460] Webserver started at http://127.3.221.65:45867/ using document root <none> and password file <none>
I20260811 11:54:08.044950 30869 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:08.045024 30869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:08.049559 30869 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:54:08.052012 30968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:08.053534 30869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:54:08.053663 30869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "87a977b6ceef49e988b94b6d5c3215fe"
format_stamp: "Formatted at 2026-08-11 11:53:25 on dist-test-slave-bndk"
server_key: "526e1c04fb107212ed2f7922088fdc89"
server_key_iv: "c5f6d42994686adb4bd9d8bace33227d"
server_key_version: "encryptionkey@0"
I20260811 11:54:08.054255 30869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:08.104960 30869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:08.109395 30869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:08.109609 30869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:08.110275 30869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:54:08.111554 30983 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260811 11:54:08.117465 30869 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 11:54:08.117552 30869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260811 11:54:08.117635 30869 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 11:54:08.119513 30869 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 11:54:08.119576 30869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260811 11:54:08.119933 30983 tablet_bootstrap.cc:492] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe: Bootstrap starting.
I20260811 11:54:08.130192 30869 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:40733
I20260811 11:54:08.130543 31155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:40733 every 8 connection(s)
I20260811 11:54:08.131366 30869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:54:08.138605 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 30869
I20260811 11:54:08.139199 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:38893
--local_ip_for_outbound_sockets=127.3.221.66
--tserver_master_addrs=127.3.221.126:36175
--webserver_port=36053
--webserver_interface=127.3.221.66
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:54:08.141630 30983 log.cc:824] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:08.152068 30983 tablet_bootstrap.cc:492] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 11:54:08.152580 30983 tablet_bootstrap.cc:492] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe: Bootstrap complete.
I20260811 11:54:08.154045 30983 ts_tablet_manager.cc:1397] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.007s
I20260811 11:54:08.157492 30983 raft_consensus.cc:348] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } }
I20260811 11:54:08.158130 30983 raft_consensus.cc:729] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87a977b6ceef49e988b94b6d5c3215fe, State: Initialized, Role: FOLLOWER
I20260811 11:54:08.158772 30983 consensus_queue.cc:260] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } }
I20260811 11:54:08.159925 30983 ts_tablet_manager.cc:1428] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260811 11:54:08.186264 31159 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:54:08.133858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.65:46255)
0811 11:54:08.134473 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:08.134537 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:08.135723 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:08.185247 (+ 49524us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":150,"threads_started":1}
W20260811 11:54:08.186990 31156 heartbeater.cc:644] Failed to heartbeat to 127.3.221.126:36175 (0 consecutive failures): Network error: Failed to ping master at 127.3.221.126:36175: Client connection negotiation failed: client connection to 127.3.221.126:36175: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260811 11:54:08.187791 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:08.187503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.65:59665)
0811 11:54:08.187640 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:08.187719 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260811 11:54:08.188524 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:08.188251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.65:35691)
0811 11:54:08.188391 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:08.188449 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260811 11:54:08.188740 31156 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 11:54:08.267663 31162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:08.267989 31162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:08.268056 31162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:08.272266 31162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:08.272356 31162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:08.272418 31162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:54:08.278127 31162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:38893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=36053
--tserver_master_addrs=127.3.221.126:36175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:08.279352 31162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:08.280390 31162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:08.291142 31178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:08.291271 31162 server_base.cc:1034] running on GCE node
W20260811 11:54:08.291142 31177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:08.291142 31181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:08.291911 31162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:08.292593 31162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:08.293769 31162 hybrid_clock.cc:648] HybridClock initialized: now 1786449248293741 us; error 43 us; skew 500 ppm
Aug 11 11:54:08 dist-test-slave-bndk krb5kdc[29033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:08.296725 31162 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:54:08.297763 31162 webserver.cc:460] Webserver started at http://127.3.221.66:36053/ using document root <none> and password file <none>
I20260811 11:54:08.298349 31162 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:08.298434 31162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:08.302556 31162 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260811 11:54:08.305162 31190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:08.306988 31162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 11:54:08.307154 31162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "0642a3ad98fe46869269ca187bd2d5fd"
format_stamp: "Formatted at 2026-08-11 11:53:25 on dist-test-slave-bndk"
server_key: "d9695daf9862f39a7446a81ef5768419"
server_key_iv: "b77e0c71a86320dba523b9ae71d57ca0"
server_key_version: "encryptionkey@0"
I20260811 11:54:08.307928 31162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:08.333284 31162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:08.336822 31162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:08.337041 31162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:08.337749 31162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:54:08.338676 31162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:54:08.338727 31162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:08.338785 31162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:54:08.338814 31162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:08.348671 31162 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:38893
I20260811 11:54:08.348695 31361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:38893 every 8 connection(s)
I20260811 11:54:08.349944 31162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:54:08.354059 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:08.352301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.66:42935)
0811 11:54:08.352810 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:08.353293 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":123,"threads_started":1}
W20260811 11:54:08.354628 31365 heartbeater.cc:644] Failed to heartbeat to 127.3.221.126:36175 (0 consecutive failures): Network error: Failed to ping master at 127.3.221.126:36175: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
I20260811 11:54:08.355300 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:08.355070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.66:58219)
0811 11:54:08.355189 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:08.355242 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260811 11:54:08.355988 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:08.355791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.66:58577)
0811 11:54:08.355887 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:08.355929 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260811 11:54:08.356144 31365 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 11:54:08.360162 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 31162
I20260811 11:54:08.360600 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:46597
--local_ip_for_outbound_sockets=127.3.221.67
--tserver_master_addrs=127.3.221.126:36175
--webserver_port=39321
--webserver_interface=127.3.221.67
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:08.482026 31368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:08.482359 31368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:08.482407 31368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:08.486989 31368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:08.487078 31368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:08.487157 31368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:54:08.493171 31368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:46597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=39321
--tserver_master_addrs=127.3.221.126:36175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:08.494545 31368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:08.495683 31368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:08.505872 31376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:08.505872 31378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:08.505882 31382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:08.506232 31368 server_base.cc:1034] running on GCE node
I20260811 11:54:08.507098 31368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:08.507915 31368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:08.509109 31368 hybrid_clock.cc:648] HybridClock initialized: now 1786449248509085 us; error 38 us; skew 500 ppm
Aug 11 11:54:08 dist-test-slave-bndk krb5kdc[29033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:08.512107 31368 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:54:08.513064 31368 webserver.cc:460] Webserver started at http://127.3.221.67:39321/ using document root <none> and password file <none>
I20260811 11:54:08.513681 31368 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:08.513751 31368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:08.517879 31368 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 11:54:08.520221 31399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:08.521696 31368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:54:08.521821 31368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "dc6745c5de9b426690d9e3ec5ae80bb9"
format_stamp: "Formatted at 2026-08-11 11:53:25 on dist-test-slave-bndk"
server_key: "23dbd358c7ce80ac09e529116902bb89"
server_key_iv: "c8b93a517fd4bbcdf50f1a4c2ff8e07f"
server_key_version: "encryptionkey@0"
I20260811 11:54:08.522433 31368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:08.549536 31368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:08.553046 31368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:08.553272 31368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:08.553939 31368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:54:08.555190 31368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:54:08.555256 31368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:08.555325 31368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:54:08.555354 31368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:08.565433 31368 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:46597
I20260811 11:54:08.565474 31589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:46597 every 8 connection(s)
I20260811 11:54:08.566545 31368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:54:08.569568 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 31368
I20260811 11:54:08.570984 31594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:08.568819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.67:56387)
0811 11:54:08.569406 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:08.569993 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":124,"threads_started":1}
W20260811 11:54:08.571583 31590 heartbeater.cc:644] Failed to heartbeat to 127.3.221.126:36175 (0 consecutive failures): Network error: Failed to ping master at 127.3.221.126:36175: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
I20260811 11:54:08.572341 31594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:08.572052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.67:41169)
0811 11:54:08.572200 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:08.572254 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260811 11:54:08.573323 31594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:08.572984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.67:41121)
0811 11:54:08.573111 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:08.573161 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260811 11:54:08.573530 31590 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 11:54:09.190099 31597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:09.189600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.65:60669)
0811 11:54:09.189889 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:09.189973 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
I20260811 11:54:09.357127 31598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:09.356709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.66:57443)
0811 11:54:09.356958 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:09.357022 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":75,"threads_started":1}
I20260811 11:54:09.574697 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:09.574144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.67:56453)
0811 11:54:09.574515 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:09.574575 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":87,"threads_started":1}
I20260811 11:54:10.007406 31600 raft_consensus.cc:482] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:54:10.007652 31600 raft_consensus.cc:504] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } }
I20260811 11:54:10.008757 31600 leader_election.cc:288] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0642a3ad98fe46869269ca187bd2d5fd (127.3.221.66:38893), dc6745c5de9b426690d9e3ec5ae80bb9 (127.3.221.67:46597)
Aug 11 11:54:10 dist-test-slave-bndk krb5kdc[29033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.66@KRBTEST.COM
Aug 11 11:54:10 dist-test-slave-bndk krb5kdc[29033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.67@KRBTEST.COM
I20260811 11:54:10.028646 31607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:10.009356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:58797 (local address 127.3.221.67:46597)
0811 11:54:10.009677 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:10.009696 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:10.009728 (+ 32us) server_negotiation.cc:407] Connection header received
0811 11:54:10.010396 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:10.010402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:10.010701 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:10.010986 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:10.012427 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:10.013500 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:10.014806 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:10.015268 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:10.015425 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:10.023128 (+ 7703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:10.023135 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:10.023150 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:10.023188 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:10.025176 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:10.026032 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:10.026040 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:10.026045 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:10.026215 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:10.026648 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:10.026651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:10.026653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:10.027248 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:10.027390 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:10.028141 (+ 751us) server_negotiation.cc:299] Negotiation successful
0811 11:54:10.028455 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":114,"threads_started":1}
I20260811 11:54:10.028635 31602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:10.008989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:38893 (local address 127.3.221.65:56161)
0811 11:54:10.009314 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:10.009330 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:10.009443 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:10.011087 (+ 1644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:10.011094 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:10.011706 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:54:10.012240 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:10.012256 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:10.013645 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:10.013647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:10.014666 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:10.014677 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:10.015383 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:10.015386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:10.015593 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:10.020661 (+ 5068us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:54:10.020687 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:54:10.022574 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:54:10.025365 (+ 2791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:54:10.025373 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:54:10.025390 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:54:10.025908 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:54:10.026310 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:54:10.026313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:54:10.026315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:54:10.026548 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:54:10.027395 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:54:10.027403 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:54:10.027762 (+ 359us) client_negotiation.cc:769] Sending connection context
0811 11:54:10.027994 (+ 232us) client_negotiation.cc:240] Negotiation successful
0811 11:54:10.028395 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":271,"spinlock_wait_cycles":57216,"thread_start_us":93,"threads_started":1}
I20260811 11:54:10.028635 31603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:10.008989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:46597 (local address 127.3.221.65:58797)
0811 11:54:10.009537 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:10.009550 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:10.009663 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:10.011087 (+ 1424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:10.011094 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:10.011706 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:54:10.012242 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:10.012258 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:10.013646 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:10.013649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:10.014668 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:10.014678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:10.015384 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:10.015388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:10.015620 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:10.020662 (+ 5042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:54:10.020689 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:54:10.022894 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:54:10.025365 (+ 2471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:54:10.025373 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:54:10.025390 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:54:10.025908 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:54:10.026310 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:54:10.026313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:54:10.026315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:54:10.026541 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:54:10.027395 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:54:10.027403 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:54:10.027762 (+ 359us) client_negotiation.cc:769] Sending connection context
0811 11:54:10.027994 (+ 232us) client_negotiation.cc:240] Negotiation successful
0811 11:54:10.028464 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":405,"spinlock_wait_cycles":38912,"thread_start_us":65,"threads_started":1}
I20260811 11:54:10.029316 31606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:10.009138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:56161 (local address 127.3.221.66:38893)
0811 11:54:10.009496 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:10.009512 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:10.009561 (+ 49us) server_negotiation.cc:407] Connection header received
0811 11:54:10.010314 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:10.010321 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:10.010686 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:10.010986 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:10.012427 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:10.013500 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:10.014791 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:10.015263 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:10.015417 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:10.022869 (+ 7452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:10.022876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:10.022892 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:10.022930 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:10.025176 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:10.026044 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:10.026050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:10.026054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:10.026215 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:10.026648 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:10.026651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:10.026653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:10.027248 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:10.027390 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:10.028141 (+ 751us) server_negotiation.cc:299] Negotiation successful
0811 11:54:10.028480 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":116,"threads_started":1}
W20260811 11:54:10.032004 30977 leader_election.cc:341] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc6745c5de9b426690d9e3ec5ae80bb9 (127.3.221.67:46597): Not found: Tablet not found: 6141b6509ab64fdb83b200879d70da6e
W20260811 11:54:10.032547 30972 leader_election.cc:341] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0642a3ad98fe46869269ca187bd2d5fd (127.3.221.66:38893): Not found: Tablet not found: 6141b6509ab64fdb83b200879d70da6e
I20260811 11:54:10.032711 30972 leader_election.cc:302] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87a977b6ceef49e988b94b6d5c3215fe; no voters: 0642a3ad98fe46869269ca187bd2d5fd, dc6745c5de9b426690d9e3ec5ae80bb9
I20260811 11:54:10.033012 31600 raft_consensus.cc:2726] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:54:10.191394 31603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:10.191048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.65:51129)
0811 11:54:10.191228 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:10.191301 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260811 11:54:10.358520 31608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:10.357933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.66:60941)
0811 11:54:10.358272 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:10.358358 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":112,"threads_started":1}
I20260811 11:54:10.576153 31610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:10.575722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.67:39763)
0811 11:54:10.575988 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:10.576050 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260811 11:54:11.192795 31611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:11.192295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.65:35977)
0811 11:54:11.192619 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:11.192689 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":87,"threads_started":1}
I20260811 11:54:11.359958 31613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:11.359471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.66:52533)
0811 11:54:11.359785 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:11.359854 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":101,"threads_started":1}
I20260811 11:54:11.577533 31614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:11.577020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.67:34753)
0811 11:54:11.577358 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:11.577417 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":79,"threads_started":1}
I20260811 11:54:11.835021 31616 raft_consensus.cc:482] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:54:11.835163 31616 raft_consensus.cc:504] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } }
I20260811 11:54:11.835621 31616 leader_election.cc:288] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0642a3ad98fe46869269ca187bd2d5fd (127.3.221.66:38893), dc6745c5de9b426690d9e3ec5ae80bb9 (127.3.221.67:46597)
W20260811 11:54:11.836445 30977 leader_election.cc:341] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc6745c5de9b426690d9e3ec5ae80bb9 (127.3.221.67:46597): Not found: Tablet not found: 6141b6509ab64fdb83b200879d70da6e
W20260811 11:54:11.836588 30972 leader_election.cc:341] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0642a3ad98fe46869269ca187bd2d5fd (127.3.221.66:38893): Not found: Tablet not found: 6141b6509ab64fdb83b200879d70da6e
I20260811 11:54:11.836683 30972 leader_election.cc:302] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87a977b6ceef49e988b94b6d5c3215fe; no voters: 0642a3ad98fe46869269ca187bd2d5fd, dc6745c5de9b426690d9e3ec5ae80bb9
I20260811 11:54:11.836836 31616 raft_consensus.cc:2726] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:54:12.194237 31618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:12.193701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.65:33417)
0811 11:54:12.194060 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:12.194123 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":155,"threads_started":1}
I20260811 11:54:12.361286 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:12.360791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.66:60961)
0811 11:54:12.361068 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:12.361133 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":80,"threads_started":1}
I20260811 11:54:12.578907 31622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:12.578422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.67:54221)
0811 11:54:12.578738 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:12.578804 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":84,"threads_started":1}
I20260811 11:54:13.195858 31623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:13.195351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.65:39199)
0811 11:54:13.195686 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:13.195748 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":99,"threads_started":1}
I20260811 11:54:13.362627 31625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:13.362165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.66:58817)
0811 11:54:13.362459 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:13.362525 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":103,"threads_started":1}
I20260811 11:54:13.580361 31629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:13.579837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.67:43259)
0811 11:54:13.580161 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:13.580229 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":91,"threads_started":1}
I20260811 11:54:14.197459 31631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:14.196891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.65:42621)
0811 11:54:14.197267 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:14.197330 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":102,"threads_started":1}
I20260811 11:54:14.364452 31633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:14.363870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.66:43803)
0811 11:54:14.364268 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:14.364344 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":112,"threads_started":1}
I20260811 11:54:14.581802 31634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:14.581328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.67:34383)
0811 11:54:14.581635 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:14.581699 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":82,"threads_started":1}
I20260811 11:54:15.140151 31636 raft_consensus.cc:482] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:54:15.140318 31636 raft_consensus.cc:504] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } }
I20260811 11:54:15.140772 31636 leader_election.cc:288] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0642a3ad98fe46869269ca187bd2d5fd (127.3.221.66:38893), dc6745c5de9b426690d9e3ec5ae80bb9 (127.3.221.67:46597)
W20260811 11:54:15.141754 30972 leader_election.cc:341] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0642a3ad98fe46869269ca187bd2d5fd (127.3.221.66:38893): Not found: Tablet not found: 6141b6509ab64fdb83b200879d70da6e
W20260811 11:54:15.141878 30977 leader_election.cc:341] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc6745c5de9b426690d9e3ec5ae80bb9 (127.3.221.67:46597): Not found: Tablet not found: 6141b6509ab64fdb83b200879d70da6e
I20260811 11:54:15.141942 30977 leader_election.cc:302] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87a977b6ceef49e988b94b6d5c3215fe; no voters: 0642a3ad98fe46869269ca187bd2d5fd, dc6745c5de9b426690d9e3ec5ae80bb9
I20260811 11:54:15.142092 31636 raft_consensus.cc:2726] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:54:15.199067 31637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:15.198438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.65:47883)
0811 11:54:15.198892 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:15.198960 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":104,"threads_started":1}
I20260811 11:54:15.366029 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:15.365503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.66:35161)
0811 11:54:15.365856 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:15.365923 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":80,"threads_started":1}
I20260811 11:54:15.583262 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:15.582679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.67:34629)
0811 11:54:15.583083 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:15.583153 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":101,"threads_started":1}
I20260811 11:54:16.200951 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:16.200279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.65:36163)
0811 11:54:16.200745 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:16.200831 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":155,"threads_started":1}
I20260811 11:54:16.367584 31646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:16.367065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.66:47495)
0811 11:54:16.367385 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:16.367457 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":79,"threads_started":1}
I20260811 11:54:16.584777 31647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:16.584222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.67:46065)
0811 11:54:16.584607 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:16.584669 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":170,"threads_started":1}
I20260811 11:54:17.202508 31648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:17.201999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.65:59607)
0811 11:54:17.202332 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:17.202401 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":88,"threads_started":1}
I20260811 11:54:17.369016 31650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:17.368514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.66:33109)
0811 11:54:17.368848 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:17.368913 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":94,"threads_started":1}
I20260811 11:54:17.586532 31651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:17.585830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.67:41577)
0811 11:54:17.586171 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:17.586238 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":110,"threads_started":1}
I20260811 11:54:18.204097 31652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:18.203589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.65:56175)
0811 11:54:18.203923 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:18.203986 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":99,"threads_started":1}
I20260811 11:54:18.370586 31655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:18.370085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.66:54931)
0811 11:54:18.370404 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:18.370475 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":85,"threads_started":1}
I20260811 11:54:18.588425 31656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:18.587897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.67:37261)
0811 11:54:18.588238 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:18.588306 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":90,"threads_started":1}
I20260811 11:54:19.205552 31657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:19.205029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.65:49975)
0811 11:54:19.205376 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:19.205444 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":92,"threads_started":1}
I20260811 11:54:19.215337 31658 raft_consensus.cc:482] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:54:19.215490 31658 raft_consensus.cc:504] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0642a3ad98fe46869269ca187bd2d5fd" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 38893 } } peers { permanent_uuid: "dc6745c5de9b426690d9e3ec5ae80bb9" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 46597 } } peers { permanent_uuid: "87a977b6ceef49e988b94b6d5c3215fe" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 40733 } }
I20260811 11:54:19.215952 31658 leader_election.cc:288] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0642a3ad98fe46869269ca187bd2d5fd (127.3.221.66:38893), dc6745c5de9b426690d9e3ec5ae80bb9 (127.3.221.67:46597)
W20260811 11:54:19.216727 30972 leader_election.cc:341] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0642a3ad98fe46869269ca187bd2d5fd (127.3.221.66:38893): Not found: Tablet not found: 6141b6509ab64fdb83b200879d70da6e
W20260811 11:54:19.217106 30977 leader_election.cc:341] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc6745c5de9b426690d9e3ec5ae80bb9 (127.3.221.67:46597): Not found: Tablet not found: 6141b6509ab64fdb83b200879d70da6e
I20260811 11:54:19.217198 30977 leader_election.cc:302] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87a977b6ceef49e988b94b6d5c3215fe; no voters: 0642a3ad98fe46869269ca187bd2d5fd, dc6745c5de9b426690d9e3ec5ae80bb9
I20260811 11:54:19.217424 31658 raft_consensus.cc:2726] T 6141b6509ab64fdb83b200879d70da6e P 87a977b6ceef49e988b94b6d5c3215fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 11:54:19.372447 31659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:19.371745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.66:40609)
0811 11:54:19.372217 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:19.372305 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":134,"threads_started":1}
I20260811 11:54:19.590205 31661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:19.589711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.67:55877)
0811 11:54:19.590035 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:19.590103 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":105,"threads_started":1}
I20260811 11:54:20.207046 31662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:20.206561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.65:35219)
0811 11:54:20.206876 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:20.206942 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":88,"threads_started":1}
I20260811 11:54:20.373997 31663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:20.373458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.66:36993)
0811 11:54:20.373812 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:20.373885 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":85,"threads_started":1}
I20260811 11:54:20.591667 31664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:20.591140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.67:41567)
0811 11:54:20.591507 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:20.591568 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":113,"threads_started":1}
I20260811 11:54:21.208910 31665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:21.208202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.65:53807)
0811 11:54:21.208642 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:21.208758 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":96,"threads_started":1}
I20260811 11:54:21.375756 31666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:21.375055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.66:54107)
0811 11:54:21.375526 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:21.375617 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":119,"threads_started":1}
I20260811 11:54:21.593421 31667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:21.592648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.67:38267)
0811 11:54:21.593160 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:21.593301 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":126,"threads_started":1}
I20260811 11:54:22.210696 31668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:22.210088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.65:55637)
0811 11:54:22.210402 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:22.210560 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":105,"threads_started":1}
I20260811 11:54:22.377379 31669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:22.376786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.66:59343)
0811 11:54:22.377135 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:22.377205 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":94,"threads_started":1}
I20260811 11:54:22.594897 31670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:22.594391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.67:42059)
0811 11:54:22.594711 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:22.594780 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":91,"threads_started":1}
I20260811 11:54:23.213016 31673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:23.212288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.65:33341)
0811 11:54:23.212797 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:23.212896 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":129,"threads_started":1}
I20260811 11:54:23.379072 31674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:23.378592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.66:33631)
0811 11:54:23.378892 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:23.378968 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":90,"threads_started":1}
I20260811 11:54:23.573089 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 30869
I20260811 11:54:23.596589 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 31162
I20260811 11:54:23.599502 31676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:23.596253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36175 (local address 127.3.221.67:57557)
0811 11:54:23.596616 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:23.596672 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.126:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":83,"threads_started":1}
I20260811 11:54:23.612458 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 31368
2026-08-11T11:54:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (60082 ms)
[----------] 30 tests from SecurityITest (259474 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:54:23 dist-test-slave-bndk krb5kdc[31682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:54:23 dist-test-slave-bndk krb5kdc[31682](info): set up 2 sockets
Aug 11 11:54:23 dist-test-slave-bndk krb5kdc[31682](info): commencing operation
krb5kdc: starting...
W20260811 11:54:24.868316 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:54:24 dist-test-slave-bndk krb5kdc[31682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:54:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:54:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:54:25.004591 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:44677
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:44677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:25.130396 31698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:25.130720 31698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:25.130764 31698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:25.134990 31698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:54:25.135078 31698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:25.135099 31698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:25.135115 31698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:54:25.135130 31698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:54:25.140899 31698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:44677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:44677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:25.142439 31698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:25.144181 31698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:54:25.154677 31698 server_base.cc:1034] running on GCE node
W20260811 11:54:25.154644 31710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:25.154629 31706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:25.154633 31707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:25.155524 31698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:25.156781 31698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:25.157982 31698 hybrid_clock.cc:648] HybridClock initialized: now 1786449265157966 us; error 30 us; skew 500 ppm
Aug 11 11:54:25 dist-test-slave-bndk krb5kdc[31682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449265, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:25.161145 31698 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:54:25.162217 31698 webserver.cc:460] Webserver started at http://127.3.221.126:41735/ using document root <none> and password file <none>
I20260811 11:54:25.162838 31698 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:25.162910 31698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:25.163130 31698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:25.165515 31698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "4fc973172d164de0b390650483af5d36"
format_stamp: "Formatted at 2026-08-11 11:54:25 on dist-test-slave-bndk"
server_key: "d1515445b0fea2a213af8888c8059ef2"
server_key_iv: "b90badcd73b3a801efaff9defa5c9cc3"
server_key_version: "encryptionkey@0"
I20260811 11:54:25.166189 31698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "4fc973172d164de0b390650483af5d36"
format_stamp: "Formatted at 2026-08-11 11:54:25 on dist-test-slave-bndk"
server_key: "d1515445b0fea2a213af8888c8059ef2"
server_key_iv: "b90badcd73b3a801efaff9defa5c9cc3"
server_key_version: "encryptionkey@0"
I20260811 11:54:25.170989 31698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260811 11:54:25.174145 31719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:25.175652 31698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 11:54:25.175797 31698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "4fc973172d164de0b390650483af5d36"
format_stamp: "Formatted at 2026-08-11 11:54:25 on dist-test-slave-bndk"
server_key: "d1515445b0fea2a213af8888c8059ef2"
server_key_iv: "b90badcd73b3a801efaff9defa5c9cc3"
server_key_version: "encryptionkey@0"
I20260811 11:54:25.175920 31698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:25.192783 31698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:25.196698 31698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:25.196890 31698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:25.204484 31698 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:44677
I20260811 11:54:25.204578 31805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:44677 every 8 connection(s)
I20260811 11:54:25.205821 31698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:54:25.209035 31806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:25.214205 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 31698
I20260811 11:54:25.214346 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:54:25.214665 3957 external_mini_cluster.cc:1428] Setting key fb7b7e6f9ad488883985a2a2e22fb4d8
I20260811 11:54:25.215308 31806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36: Bootstrap starting.
I20260811 11:54:25.218236 31806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:25.219045 31806 log.cc:824] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:25.221115 31806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36: No bootstrap required, opened a new log
Aug 11 11:54:25 dist-test-slave-bndk krb5kdc[31682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:54:25.225723 31806 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc973172d164de0b390650483af5d36" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 44677 } }
I20260811 11:54:25.226073 31806 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:25.226148 31806 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc973172d164de0b390650483af5d36, State: Initialized, Role: FOLLOWER
I20260811 11:54:25.226694 31806 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc973172d164de0b390650483af5d36" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 44677 } }
I20260811 11:54:25.226876 31806 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:54:25.226948 31806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:54:25.227054 31806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:25.228374 31806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc973172d164de0b390650483af5d36" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 44677 } }
I20260811 11:54:25.228816 31806 leader_election.cc:302] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fc973172d164de0b390650483af5d36; no voters:
I20260811 11:54:25.229452 31806 leader_election.cc:288] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:54:25.229511 31811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:25.229741 31811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36 [term 1 LEADER]: Becoming Leader. State: Replica: 4fc973172d164de0b390650483af5d36, State: Running, Role: LEADER
I20260811 11:54:25.230093 31811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc973172d164de0b390650483af5d36" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 44677 } }
I20260811 11:54:25.230928 31809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:25.216254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39832 (local address 127.3.221.126:44677)
0811 11:54:25.216876 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:25.216895 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:25.216935 (+ 40us) server_negotiation.cc:407] Connection header received
0811 11:54:25.217694 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:25.217714 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:25.218077 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:25.218400 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:25.219225 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:25.220361 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:25.221331 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:25.222029 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:25.222220 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:25.225163 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:25.225170 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:25.225193 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:25.225261 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:25.227321 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:25.227799 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:25.227805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:25.227811 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:25.227977 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:25.228303 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:25.228307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:25.228309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:25.228927 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:25.229273 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:25.229473 (+ 200us) server_negotiation.cc:299] Negotiation successful
0811 11:54:25.229909 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":170,"threads_started":1}
I20260811 11:54:25.230981 31806 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:54:25.231276 31811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fc973172d164de0b390650483af5d36. Latest consensus state: current_term: 1 leader_uuid: "4fc973172d164de0b390650483af5d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc973172d164de0b390650483af5d36" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 44677 } } }
I20260811 11:54:25.231402 31811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:25.231621 31811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fc973172d164de0b390650483af5d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc973172d164de0b390650483af5d36" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 44677 } } }
I20260811 11:54:25.231702 31811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:25.231875 31816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:54:25.235517 31816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:54:25.242858 31816 catalog_manager.cc:1347] Generated new cluster ID: 55c2f8382e0f4986a90523bdcd328508
I20260811 11:54:25.242954 31816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:54:25.262458 31816 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:54:25.263497 31816 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:54:25.270802 31816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36: Generated new TSK 0
I20260811 11:54:25.272023 31816 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
I20260811 11:54:25.308264 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:44677
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:25.431454 31840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:25.431783 31840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:25.431841 31840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:25.436098 31840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:25.436167 31840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:25.436231 31840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:54:25.441816 31840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:44677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:25.442930 31840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:25.443954 31840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:54:25.454066 31840 server_base.cc:1034] running on GCE node
W20260811 11:54:25.453998 31846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:25.454018 31850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:25.454034 31845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:25.454730 31840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:25.455389 31840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:25.456583 31840 hybrid_clock.cc:648] HybridClock initialized: now 1786449265456560 us; error 34 us; skew 500 ppm
Aug 11 11:54:25 dist-test-slave-bndk krb5kdc[31682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449265, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:25.459611 31840 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:54:25.460609 31840 webserver.cc:460] Webserver started at http://127.3.221.65:41555/ using document root <none> and password file <none>
I20260811 11:54:25.461189 31840 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:25.461310 31840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:25.461537 31840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:25.463729 31840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "8c8f4549bec64519aaae94f3c2bf4514"
format_stamp: "Formatted at 2026-08-11 11:54:25 on dist-test-slave-bndk"
server_key: "fcee452e1c6f01d5e6f007beec71c5c0"
server_key_iv: "d9c39a42b2d3b293a17c877b3b15b0e3"
server_key_version: "encryptionkey@0"
I20260811 11:54:25.464332 31840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "8c8f4549bec64519aaae94f3c2bf4514"
format_stamp: "Formatted at 2026-08-11 11:54:25 on dist-test-slave-bndk"
server_key: "fcee452e1c6f01d5e6f007beec71c5c0"
server_key_iv: "d9c39a42b2d3b293a17c877b3b15b0e3"
server_key_version: "encryptionkey@0"
I20260811 11:54:25.468639 31840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260811 11:54:25.471481 31865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:25.472826 31840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 11:54:25.472957 31840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "8c8f4549bec64519aaae94f3c2bf4514"
format_stamp: "Formatted at 2026-08-11 11:54:25 on dist-test-slave-bndk"
server_key: "fcee452e1c6f01d5e6f007beec71c5c0"
server_key_iv: "d9c39a42b2d3b293a17c877b3b15b0e3"
server_key_version: "encryptionkey@0"
I20260811 11:54:25.473073 31840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:25.500363 31840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:25.503894 31840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:25.504101 31840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:25.504782 31840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:54:25.505715 31840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:54:25.505769 31840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:25.505825 31840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:54:25.505851 31840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:25.515094 31840 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:45691
I20260811 11:54:25.515126 32044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:45691 every 8 connection(s)
I20260811 11:54:25.516091 31840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:54:25.516849 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 31840
I20260811 11:54:25.516990 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:54:25.517342 3957 external_mini_cluster.cc:1428] Setting key d6c46f0436452bffccda2d94c65befea
Aug 11 11:54:25 dist-test-slave-bndk krb5kdc[31682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449265, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.66@KRBTEST.COM" created.
I20260811 11:54:25.531479 31809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:25.518181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:44527 (local address 127.3.221.126:44677)
0811 11:54:25.518327 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:25.518332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:25.519431 (+ 1099us) server_negotiation.cc:407] Connection header received
0811 11:54:25.520522 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:25.520525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:25.520571 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:25.520641 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:25.522171 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:25.522726 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:25.523995 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:25.524236 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:25.524329 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:25.527210 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:25.527213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:25.527215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:25.527249 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:25.528878 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:25.529615 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:25.529618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:25.529619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:25.529724 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:25.530170 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:25.530173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:25.530175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:25.530394 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:25.530478 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:25.531167 (+ 689us) server_negotiation.cc:299] Negotiation successful
0811 11:54:25.531340 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 11:54:25.532673 32050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:25.518660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:44677 (local address 127.3.221.65:44527)
0811 11:54:25.519250 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:25.519312 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:25.520289 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:25.520829 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:25.520838 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:25.521361 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:54:25.521985 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:25.522005 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:25.522861 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:25.522863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:25.523853 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:25.523865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:25.524395 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:25.524400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:25.524588 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:25.525192 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:54:25.525244 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:54:25.526970 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:54:25.529030 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:54:25.529037 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:54:25.529055 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:54:25.529489 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:54:25.529861 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:54:25.529864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:54:25.529866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:54:25.530045 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:54:25.530482 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:54:25.530488 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:54:25.530848 (+ 360us) client_negotiation.cc:769] Sending connection context
0811 11:54:25.531095 (+ 247us) client_negotiation.cc:240] Negotiation successful
0811 11:54:25.531535 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":142,"threads_started":1}
I20260811 11:54:25.534220 32045 heartbeater.cc:347] Connected to a master server at 127.3.221.126:44677
I20260811 11:54:25.534499 32045 heartbeater.cc:464] Registering TS with master...
I20260811 11:54:25.535068 32045 heartbeater.cc:511] Master 127.3.221.126:44677 requested a full tablet report, sending...
I20260811 11:54:25.536959 31750 ts_manager.cc:194] Registered new tserver with Master: 8c8f4549bec64519aaae94f3c2bf4514 (127.3.221.65:45691)
I20260811 11:54:25.538200 31750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:44527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
I20260811 11:54:25.550271 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:44677
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:25.668643 32054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:25.668936 32054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:25.668983 32054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:25.673149 32054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:25.673271 32054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:25.673341 32054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:54:25.679157 32054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:44677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:25.680349 32054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:25.681499 32054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:25.691756 32066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:25.691792 32067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:25.692198 32070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:25.692425 32054 server_base.cc:1034] running on GCE node
I20260811 11:54:25.692950 32054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:25.693876 32054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:25.695109 32054 hybrid_clock.cc:648] HybridClock initialized: now 1786449265695070 us; error 43 us; skew 500 ppm
Aug 11 11:54:25 dist-test-slave-bndk krb5kdc[31682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449265, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:25.699332 32054 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:54:25.700573 32054 webserver.cc:460] Webserver started at http://127.3.221.66:32879/ using document root <none> and password file <none>
I20260811 11:54:25.701210 32054 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:25.701323 32054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:25.701550 32054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:25.703822 32054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "1ccebf5803594ad7b73b279a35ff1296"
format_stamp: "Formatted at 2026-08-11 11:54:25 on dist-test-slave-bndk"
server_key: "29894e0548d62538d301279702b1ef5d"
server_key_iv: "682bfc2219e3fddb9c687b01c79ff458"
server_key_version: "encryptionkey@0"
I20260811 11:54:25.704550 32054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "1ccebf5803594ad7b73b279a35ff1296"
format_stamp: "Formatted at 2026-08-11 11:54:25 on dist-test-slave-bndk"
server_key: "29894e0548d62538d301279702b1ef5d"
server_key_iv: "682bfc2219e3fddb9c687b01c79ff458"
server_key_version: "encryptionkey@0"
I20260811 11:54:25.708868 32054 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 11:54:25.711669 32081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:25.713027 32054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 11:54:25.713145 32054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "1ccebf5803594ad7b73b279a35ff1296"
format_stamp: "Formatted at 2026-08-11 11:54:25 on dist-test-slave-bndk"
server_key: "29894e0548d62538d301279702b1ef5d"
server_key_iv: "682bfc2219e3fddb9c687b01c79ff458"
server_key_version: "encryptionkey@0"
I20260811 11:54:25.713335 32054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:25.731014 32054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:25.734562 32054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:25.734760 32054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:25.735394 32054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:54:25.736280 32054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:54:25.736332 32054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:25.736375 32054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:54:25.736404 32054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:25.745205 32054 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:46155
I20260811 11:54:25.745499 32259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:46155 every 8 connection(s)
I20260811 11:54:25.746248 32054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:54:25.748767 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 32054
I20260811 11:54:25.748883 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:54:25.749172 3957 external_mini_cluster.cc:1428] Setting key 03a3642f62fc0f12f92b0dbd289bc577
Aug 11 11:54:25 dist-test-slave-bndk krb5kdc[31682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449265, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.221.67@KRBTEST.COM; defaulting to no policy
I20260811 11:54:25.762873 31809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:25.748287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:36377 (local address 127.3.221.126:44677)
0811 11:54:25.748472 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:25.748476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:25.750036 (+ 1560us) server_negotiation.cc:407] Connection header received
0811 11:54:25.751034 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:25.751036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:25.751080 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:25.751135 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:25.752724 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:25.753326 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:25.754963 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:25.755186 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:25.755280 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:25.758558 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:25.758561 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:25.758563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:25.758592 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:25.760270 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:25.760962 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:25.760967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:25.760969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:25.761077 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:25.761502 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:25.761505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:25.761506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:25.761731 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:25.761825 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:25.762607 (+ 782us) server_negotiation.cc:299] Negotiation successful
0811 11:54:25.762734 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.67@KRBTEST.COM" created.
I20260811 11:54:25.764084 32263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:25.748658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:44677 (local address 127.3.221.66:36377)
0811 11:54:25.749785 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:25.749913 (+ 128us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:25.750814 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:25.751359 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:25.751368 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:25.751856 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:54:25.752544 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:25.752565 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:25.753479 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:25.753483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:25.754790 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:25.754814 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:25.755352 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:25.755356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:25.755554 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:25.756205 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:54:25.756228 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:54:25.758266 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:54:25.760446 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:54:25.760454 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:54:25.760476 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:54:25.760849 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:54:25.761170 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:54:25.761173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:54:25.761175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:54:25.761390 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:54:25.761906 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:54:25.761914 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:54:25.762293 (+ 379us) client_negotiation.cc:769] Sending connection context
0811 11:54:25.762494 (+ 201us) client_negotiation.cc:240] Negotiation successful
0811 11:54:25.762864 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"thread_start_us":154,"threads_started":1}
I20260811 11:54:25.765646 32260 heartbeater.cc:347] Connected to a master server at 127.3.221.126:44677
I20260811 11:54:25.765906 32260 heartbeater.cc:464] Registering TS with master...
I20260811 11:54:25.766544 32260 heartbeater.cc:511] Master 127.3.221.126:44677 requested a full tablet report, sending...
I20260811 11:54:25.767844 31750 ts_manager.cc:194] Registered new tserver with Master: 1ccebf5803594ad7b73b279a35ff1296 (127.3.221.66:46155)
I20260811 11:54:25.768489 31750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:36377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
I20260811 11:54:25.780224 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:44677
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:25.900277 32267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:25.900584 32267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:25.900632 32267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:25.904860 32267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:25.904948 32267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:25.905010 32267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:54:25.910852 32267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:44677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:25.912021 32267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:25.913064 32267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:25.923270 32280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:25.923274 32284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:25.923349 32279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:25.923722 32267 server_base.cc:1034] running on GCE node
I20260811 11:54:25.924177 32267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:25.924908 32267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:25.926092 32267 hybrid_clock.cc:648] HybridClock initialized: now 1786449265926070 us; error 41 us; skew 500 ppm
Aug 11 11:54:25 dist-test-slave-bndk krb5kdc[31682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449265, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:25.929240 32267 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:54:25.930223 32267 webserver.cc:460] Webserver started at http://127.3.221.67:36547/ using document root <none> and password file <none>
I20260811 11:54:25.931087 32267 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:25.931169 32267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:25.931484 32267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:25.934049 32267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "659d316c88ed485f8e46a0e0bbedf47c"
format_stamp: "Formatted at 2026-08-11 11:54:25 on dist-test-slave-bndk"
server_key: "3b6b763444d87a5d62fcdd2de1115668"
server_key_iv: "19457cbca0fc2bf31feeae8b9e1771e0"
server_key_version: "encryptionkey@0"
I20260811 11:54:25.934633 32267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "659d316c88ed485f8e46a0e0bbedf47c"
format_stamp: "Formatted at 2026-08-11 11:54:25 on dist-test-slave-bndk"
server_key: "3b6b763444d87a5d62fcdd2de1115668"
server_key_iv: "19457cbca0fc2bf31feeae8b9e1771e0"
server_key_version: "encryptionkey@0"
I20260811 11:54:25.938699 32267 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:54:25.941431 32295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:25.942797 32267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:54:25.942922 32267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "659d316c88ed485f8e46a0e0bbedf47c"
format_stamp: "Formatted at 2026-08-11 11:54:25 on dist-test-slave-bndk"
server_key: "3b6b763444d87a5d62fcdd2de1115668"
server_key_iv: "19457cbca0fc2bf31feeae8b9e1771e0"
server_key_version: "encryptionkey@0"
I20260811 11:54:25.943051 32267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:25.962103 32267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:25.965485 32267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:25.965682 32267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:25.966308 32267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:54:25.967199 32267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:54:25.967248 32267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:25.967304 32267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:54:25.967330 32267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:25.976482 32267 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:37449
I20260811 11:54:25.976517 32480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:37449 every 8 connection(s)
I20260811 11:54:25.977506 32267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:54:25.978773 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 32267
I20260811 11:54:25.978928 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:54:25.979223 3957 external_mini_cluster.cc:1428] Setting key 11415c1e6ef2507748d6f707cb3b7c42
Aug 11 11:54:25 dist-test-slave-bndk krb5kdc[31682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449265, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:54:25.993780 31809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:25.979626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:45015 (local address 127.3.221.126:44677)
0811 11:54:25.979754 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:25.979758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:25.981023 (+ 1265us) server_negotiation.cc:407] Connection header received
0811 11:54:25.982051 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:25.982053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:25.982102 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:25.982172 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:25.983508 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:25.984057 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:25.985415 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:25.985660 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:25.985755 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:25.989407 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:25.989411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:25.989413 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:25.989448 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:25.990996 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:25.991739 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:25.991742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:25.991744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:25.991863 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:25.992397 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:25.992401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:25.992402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:25.992654 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:25.992758 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:25.993472 (+ 714us) server_negotiation.cc:299] Negotiation successful
0811 11:54:25.993615 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 11:54:25.994714 32484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:25.980290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:44677 (local address 127.3.221.67:45015)
0811 11:54:25.980838 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:25.980913 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:25.981850 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:25.982310 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:25.982317 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:25.982702 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:54:25.983219 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:25.983233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:25.984281 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:25.984285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:25.985268 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:25.985283 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:25.985999 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:25.986006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:25.986289 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:25.987119 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:54:25.987140 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:54:25.989051 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:54:25.991193 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:54:25.991200 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:54:25.991232 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:54:25.991577 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:54:25.991990 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:54:25.991993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:54:25.991995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:54:25.992252 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:54:25.992802 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:54:25.992810 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:54:25.993155 (+ 345us) client_negotiation.cc:769] Sending connection context
0811 11:54:25.993400 (+ 245us) client_negotiation.cc:240] Negotiation successful
0811 11:54:25.993815 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":149,"threads_started":1}
I20260811 11:54:25.995925 32481 heartbeater.cc:347] Connected to a master server at 127.3.221.126:44677
I20260811 11:54:25.996163 32481 heartbeater.cc:464] Registering TS with master...
I20260811 11:54:25.996680 32481 heartbeater.cc:511] Master 127.3.221.126:44677 requested a full tablet report, sending...
I20260811 11:54:25.997845 31750 ts_manager.cc:194] Registered new tserver with Master: 659d316c88ed485f8e46a0e0bbedf47c (127.3.221.67:37449)
I20260811 11:54:25.998854 31750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:45015
I20260811 11:54:26.005609 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:54:26.015743 31809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:26.007729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39838 (local address 127.3.221.126:44677)
0811 11:54:26.007879 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:26.007882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:26.007992 (+ 110us) server_negotiation.cc:407] Connection header received
0811 11:54:26.008142 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:26.008145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:26.008195 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:26.008279 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:26.008974 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.009599 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.010494 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.010722 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.010812 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:26.012208 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:26.012212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:26.012214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:26.012263 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:26.013846 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:26.014323 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:26.014326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:26.014327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:26.014431 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:26.014776 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:26.014779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:26.014780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:26.015057 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:26.015177 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:26.015471 (+ 294us) server_negotiation.cc:299] Negotiation successful
0811 11:54:26.015600 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 11:54:26.018901 31750 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:54:26.022126 31750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:54:26.038043 32506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:26.032257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42118 (local address 127.3.221.66:46155)
0811 11:54:26.032686 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:26.032700 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:26.032762 (+ 62us) server_negotiation.cc:407] Connection header received
0811 11:54:26.032844 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:26.032849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:26.033075 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:26.033426 (+ 351us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:26.033713 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.034976 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.036632 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.037469 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.037578 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:26.037714 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:26.037805 (+ 91us) server_negotiation.cc:299] Negotiation successful
0811 11:54:26.037904 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":108,"threads_started":1}
I20260811 11:54:26.038033 32504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:26.032149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:45691 (local address 127.0.0.1:45842)
0811 11:54:26.033031 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:26.033045 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:26.033179 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:26.033832 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:26.033835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:26.033855 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:26.033932 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:26.033938 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.034888 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.034892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:26.036163 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:26.036173 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.037402 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.037407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:26.037522 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:26.037553 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 11:54:26.037659 (+ 106us) client_negotiation.cc:240] Negotiation successful
0811 11:54:26.037845 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":3,"thread_start_us":113,"threads_started":1}
I20260811 11:54:26.038038 32497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:26.031375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:37449 (local address 127.0.0.1:50238)
0811 11:54:26.031945 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:26.031996 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:26.032176 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:26.033311 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:26.033318 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:26.033360 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:26.033565 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:26.033573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.035134 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.035137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:26.036424 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:26.036437 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.037308 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.037312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:26.037493 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:26.037547 (+ 54us) client_negotiation.cc:769] Sending connection context
0811 11:54:26.037666 (+ 119us) client_negotiation.cc:240] Negotiation successful
0811 11:54:26.037861 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":149,"spinlock_wait_cycles":8064,"thread_start_us":119,"threads_started":1}
I20260811 11:54:26.038558 32507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:26.032628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.3.221.65:45691)
0811 11:54:26.033054 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:26.033069 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:26.033168 (+ 99us) server_negotiation.cc:407] Connection header received
0811 11:54:26.033321 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:26.033327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:26.033709 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:26.033859 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:26.034029 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.034764 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.036287 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.037022 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.037129 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:26.037333 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:26.038310 (+ 977us) server_negotiation.cc:299] Negotiation successful
0811 11:54:26.038415 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":74,"threads_started":1}
I20260811 11:54:26.038647 32501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:26.031620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50238 (local address 127.3.221.67:37449)
0811 11:54:26.032868 (+ 1248us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:26.032879 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:26.032899 (+ 20us) server_negotiation.cc:407] Connection header received
0811 11:54:26.032965 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:26.032969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:26.033128 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:26.033720 (+ 592us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:26.033880 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.034976 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.036579 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.037162 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.037305 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:26.037461 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:26.037877 (+ 416us) server_negotiation.cc:299] Negotiation successful
0811 11:54:26.037985 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1146,"thread_start_us":185,"threads_started":1}
I20260811 11:54:26.038826 32499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:26.031592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:46155 (local address 127.0.0.1:42118)
0811 11:54:26.032503 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:26.032518 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:26.032604 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:26.033311 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:26.033318 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:26.033358 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:26.033577 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:26.033586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.035134 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.035137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:26.036511 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:26.036519 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.037594 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.037598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:26.037687 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:26.037707 (+ 20us) client_negotiation.cc:769] Sending connection context
0811 11:54:26.037772 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 11:54:26.038063 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"mutex_wait_us":278,"spinlock_wait_cycles":6912,"thread_start_us":109,"threads_started":1}
I20260811 11:54:26.041599 32372 tablet_service.cc:1467] Processing CreateTablet for tablet c58b88ffe3be4b03a6e451ad658f8e49 (DEFAULT_TABLE table=test-table [id=f1d3ee71853140e38669767dd7bede99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:54:26.041599 31942 tablet_service.cc:1467] Processing CreateTablet for tablet c58b88ffe3be4b03a6e451ad658f8e49 (DEFAULT_TABLE table=test-table [id=f1d3ee71853140e38669767dd7bede99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:54:26.042713 31942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c58b88ffe3be4b03a6e451ad658f8e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:26.042734 32372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c58b88ffe3be4b03a6e451ad658f8e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:26.049743 32510 tablet_bootstrap.cc:492] T c58b88ffe3be4b03a6e451ad658f8e49 P 659d316c88ed485f8e46a0e0bbedf47c: Bootstrap starting.
I20260811 11:54:26.049433 32145 tablet_service.cc:1467] Processing CreateTablet for tablet c58b88ffe3be4b03a6e451ad658f8e49 (DEFAULT_TABLE table=test-table [id=f1d3ee71853140e38669767dd7bede99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:54:26.050602 32511 tablet_bootstrap.cc:492] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514: Bootstrap starting.
I20260811 11:54:26.050707 32145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c58b88ffe3be4b03a6e451ad658f8e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:26.052031 32510 tablet_bootstrap.cc:654] T c58b88ffe3be4b03a6e451ad658f8e49 P 659d316c88ed485f8e46a0e0bbedf47c: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:26.052842 32511 tablet_bootstrap.cc:654] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:26.053061 32510 log.cc:824] T c58b88ffe3be4b03a6e451ad658f8e49 P 659d316c88ed485f8e46a0e0bbedf47c: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:26.053745 32511 log.cc:824] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:26.055346 32510 tablet_bootstrap.cc:492] T c58b88ffe3be4b03a6e451ad658f8e49 P 659d316c88ed485f8e46a0e0bbedf47c: No bootstrap required, opened a new log
I20260811 11:54:26.055572 32510 ts_tablet_manager.cc:1397] T c58b88ffe3be4b03a6e451ad658f8e49 P 659d316c88ed485f8e46a0e0bbedf47c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 11:54:26.055830 32511 tablet_bootstrap.cc:492] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514: No bootstrap required, opened a new log
I20260811 11:54:26.056051 32511 ts_tablet_manager.cc:1397] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 11:54:26.058832 32515 tablet_bootstrap.cc:492] T c58b88ffe3be4b03a6e451ad658f8e49 P 1ccebf5803594ad7b73b279a35ff1296: Bootstrap starting.
I20260811 11:54:26.058749 32510 raft_consensus.cc:348] T c58b88ffe3be4b03a6e451ad658f8e49 P 659d316c88ed485f8e46a0e0bbedf47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659d316c88ed485f8e46a0e0bbedf47c" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37449 } } peers { permanent_uuid: "1ccebf5803594ad7b73b279a35ff1296" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 46155 } } peers { permanent_uuid: "8c8f4549bec64519aaae94f3c2bf4514" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45691 } }
I20260811 11:54:26.059083 32510 raft_consensus.cc:374] T c58b88ffe3be4b03a6e451ad658f8e49 P 659d316c88ed485f8e46a0e0bbedf47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:26.059139 32510 raft_consensus.cc:729] T c58b88ffe3be4b03a6e451ad658f8e49 P 659d316c88ed485f8e46a0e0bbedf47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 659d316c88ed485f8e46a0e0bbedf47c, State: Initialized, Role: FOLLOWER
I20260811 11:54:26.059664 32510 consensus_queue.cc:260] T c58b88ffe3be4b03a6e451ad658f8e49 P 659d316c88ed485f8e46a0e0bbedf47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659d316c88ed485f8e46a0e0bbedf47c" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37449 } } peers { permanent_uuid: "1ccebf5803594ad7b73b279a35ff1296" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 46155 } } peers { permanent_uuid: "8c8f4549bec64519aaae94f3c2bf4514" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45691 } }
I20260811 11:54:26.060467 32481 heartbeater.cc:503] Master 127.3.221.126:44677 was elected leader, sending a full tablet report...
I20260811 11:54:26.060585 32511 raft_consensus.cc:348] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659d316c88ed485f8e46a0e0bbedf47c" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37449 } } peers { permanent_uuid: "1ccebf5803594ad7b73b279a35ff1296" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 46155 } } peers { permanent_uuid: "8c8f4549bec64519aaae94f3c2bf4514" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45691 } }
I20260811 11:54:26.060804 32510 ts_tablet_manager.cc:1428] T c58b88ffe3be4b03a6e451ad658f8e49 P 659d316c88ed485f8e46a0e0bbedf47c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260811 11:54:26.060846 32511 raft_consensus.cc:374] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:26.060909 32511 raft_consensus.cc:729] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c8f4549bec64519aaae94f3c2bf4514, State: Initialized, Role: FOLLOWER
I20260811 11:54:26.061391 32511 consensus_queue.cc:260] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659d316c88ed485f8e46a0e0bbedf47c" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37449 } } peers { permanent_uuid: "1ccebf5803594ad7b73b279a35ff1296" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 46155 } } peers { permanent_uuid: "8c8f4549bec64519aaae94f3c2bf4514" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45691 } }
I20260811 11:54:26.062443 32511 ts_tablet_manager.cc:1428] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 11:54:26.062533 32515 tablet_bootstrap.cc:654] T c58b88ffe3be4b03a6e451ad658f8e49 P 1ccebf5803594ad7b73b279a35ff1296: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:26.062808 32045 heartbeater.cc:503] Master 127.3.221.126:44677 was elected leader, sending a full tablet report...
I20260811 11:54:26.063517 32515 log.cc:824] T c58b88ffe3be4b03a6e451ad658f8e49 P 1ccebf5803594ad7b73b279a35ff1296: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:26.066082 32515 tablet_bootstrap.cc:492] T c58b88ffe3be4b03a6e451ad658f8e49 P 1ccebf5803594ad7b73b279a35ff1296: No bootstrap required, opened a new log
I20260811 11:54:26.066311 32515 ts_tablet_manager.cc:1397] T c58b88ffe3be4b03a6e451ad658f8e49 P 1ccebf5803594ad7b73b279a35ff1296: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260811 11:54:26.069464 32515 raft_consensus.cc:348] T c58b88ffe3be4b03a6e451ad658f8e49 P 1ccebf5803594ad7b73b279a35ff1296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659d316c88ed485f8e46a0e0bbedf47c" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37449 } } peers { permanent_uuid: "1ccebf5803594ad7b73b279a35ff1296" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 46155 } } peers { permanent_uuid: "8c8f4549bec64519aaae94f3c2bf4514" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45691 } }
I20260811 11:54:26.069772 32515 raft_consensus.cc:374] T c58b88ffe3be4b03a6e451ad658f8e49 P 1ccebf5803594ad7b73b279a35ff1296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:26.069834 32515 raft_consensus.cc:729] T c58b88ffe3be4b03a6e451ad658f8e49 P 1ccebf5803594ad7b73b279a35ff1296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ccebf5803594ad7b73b279a35ff1296, State: Initialized, Role: FOLLOWER
I20260811 11:54:26.070297 32515 consensus_queue.cc:260] T c58b88ffe3be4b03a6e451ad658f8e49 P 1ccebf5803594ad7b73b279a35ff1296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659d316c88ed485f8e46a0e0bbedf47c" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37449 } } peers { permanent_uuid: "1ccebf5803594ad7b73b279a35ff1296" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 46155 } } peers { permanent_uuid: "8c8f4549bec64519aaae94f3c2bf4514" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45691 } }
I20260811 11:54:26.071115 32260 heartbeater.cc:503] Master 127.3.221.126:44677 was elected leader, sending a full tablet report...
I20260811 11:54:26.071549 32515 ts_tablet_manager.cc:1428] T c58b88ffe3be4b03a6e451ad658f8e49 P 1ccebf5803594ad7b73b279a35ff1296: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260811 11:54:26.148001 32518 raft_consensus.cc:482] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:54:26.148259 32518 raft_consensus.cc:504] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659d316c88ed485f8e46a0e0bbedf47c" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37449 } } peers { permanent_uuid: "1ccebf5803594ad7b73b279a35ff1296" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 46155 } } peers { permanent_uuid: "8c8f4549bec64519aaae94f3c2bf4514" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45691 } }
I20260811 11:54:26.149652 32518 leader_election.cc:288] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 659d316c88ed485f8e46a0e0bbedf47c (127.3.221.67:37449), 1ccebf5803594ad7b73b279a35ff1296 (127.3.221.66:46155)
I20260811 11:54:26.154922 32506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:26.150095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:39183 (local address 127.3.221.66:46155)
0811 11:54:26.150223 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:26.150227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:26.150364 (+ 137us) server_negotiation.cc:407] Connection header received
0811 11:54:26.150547 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:26.150550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:26.150603 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:26.150688 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:26.151058 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.151867 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.153678 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.154325 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.154463 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:26.154521 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:26.154718 (+ 197us) server_negotiation.cc:299] Negotiation successful
0811 11:54:26.154806 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 11:54:26.154955 32524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:26.149932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:46155 (local address 127.3.221.65:39183)
0811 11:54:26.150256 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:26.150271 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:26.150437 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:26.150707 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:26.150710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:26.150738 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:26.150950 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:26.150957 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.152019 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.152023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:26.153526 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:26.153540 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.154472 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.154476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:26.154578 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:26.154607 (+ 29us) client_negotiation.cc:769] Sending connection context
0811 11:54:26.154684 (+ 77us) client_negotiation.cc:240] Negotiation successful
0811 11:54:26.154768 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":86,"spinlock_wait_cycles":16896,"thread_start_us":123,"threads_started":1}
I20260811 11:54:26.155053 32501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:26.149932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:56785 (local address 127.3.221.67:37449)
0811 11:54:26.150067 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:26.150072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:26.150371 (+ 299us) server_negotiation.cc:407] Connection header received
0811 11:54:26.150547 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:26.150550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:26.150601 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:26.150687 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:26.151058 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.151867 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.153678 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.154350 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.154463 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:26.154521 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:26.154845 (+ 324us) server_negotiation.cc:299] Negotiation successful
0811 11:54:26.154928 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 11:54:26.155449 32526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:26.149853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:37449 (local address 127.3.221.65:56785)
0811 11:54:26.150272 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:26.150285 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:26.150437 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:26.150707 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:26.150710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:26.150732 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:26.150951 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:26.150957 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.152019 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.152023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:26.153528 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:26.153540 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.154587 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.154590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:26.154697 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:26.154722 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 11:54:26.155200 (+ 478us) client_negotiation.cc:240] Negotiation successful
0811 11:54:26.155286 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":124,"spinlock_wait_cycles":27904,"thread_start_us":147,"threads_started":1}
I20260811 11:54:26.155524 32181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c58b88ffe3be4b03a6e451ad658f8e49" candidate_uuid: "8c8f4549bec64519aaae94f3c2bf4514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ccebf5803594ad7b73b279a35ff1296" is_pre_election: true
I20260811 11:54:26.155900 32181 raft_consensus.cc:2454] T c58b88ffe3be4b03a6e451ad658f8e49 P 1ccebf5803594ad7b73b279a35ff1296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c8f4549bec64519aaae94f3c2bf4514 in term 0.
I20260811 11:54:26.156064 32406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c58b88ffe3be4b03a6e451ad658f8e49" candidate_uuid: "8c8f4549bec64519aaae94f3c2bf4514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "659d316c88ed485f8e46a0e0bbedf47c" is_pre_election: true
I20260811 11:54:26.156351 32406 raft_consensus.cc:2454] T c58b88ffe3be4b03a6e451ad658f8e49 P 659d316c88ed485f8e46a0e0bbedf47c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c8f4549bec64519aaae94f3c2bf4514 in term 0.
I20260811 11:54:26.156409 31869 leader_election.cc:302] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ccebf5803594ad7b73b279a35ff1296, 8c8f4549bec64519aaae94f3c2bf4514; no voters:
I20260811 11:54:26.156728 32518 raft_consensus.cc:2781] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:54:26.156827 32518 raft_consensus.cc:482] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:54:26.156878 32518 raft_consensus.cc:3037] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:26.157970 32518 raft_consensus.cc:504] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659d316c88ed485f8e46a0e0bbedf47c" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37449 } } peers { permanent_uuid: "1ccebf5803594ad7b73b279a35ff1296" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 46155 } } peers { permanent_uuid: "8c8f4549bec64519aaae94f3c2bf4514" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45691 } }
I20260811 11:54:26.158334 32518 leader_election.cc:288] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514 [CANDIDATE]: Term 1 election: Requested vote from peers 659d316c88ed485f8e46a0e0bbedf47c (127.3.221.67:37449), 1ccebf5803594ad7b73b279a35ff1296 (127.3.221.66:46155)
I20260811 11:54:26.158704 32406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c58b88ffe3be4b03a6e451ad658f8e49" candidate_uuid: "8c8f4549bec64519aaae94f3c2bf4514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "659d316c88ed485f8e46a0e0bbedf47c"
I20260811 11:54:26.158811 32181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c58b88ffe3be4b03a6e451ad658f8e49" candidate_uuid: "8c8f4549bec64519aaae94f3c2bf4514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ccebf5803594ad7b73b279a35ff1296"
I20260811 11:54:26.158851 32406 raft_consensus.cc:3037] T c58b88ffe3be4b03a6e451ad658f8e49 P 659d316c88ed485f8e46a0e0bbedf47c [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:26.158952 32181 raft_consensus.cc:3037] T c58b88ffe3be4b03a6e451ad658f8e49 P 1ccebf5803594ad7b73b279a35ff1296 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:26.160223 32181 raft_consensus.cc:2454] T c58b88ffe3be4b03a6e451ad658f8e49 P 1ccebf5803594ad7b73b279a35ff1296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c8f4549bec64519aaae94f3c2bf4514 in term 1.
I20260811 11:54:26.160223 32406 raft_consensus.cc:2454] T c58b88ffe3be4b03a6e451ad658f8e49 P 659d316c88ed485f8e46a0e0bbedf47c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c8f4549bec64519aaae94f3c2bf4514 in term 1.
I20260811 11:54:26.160579 31868 leader_election.cc:302] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 659d316c88ed485f8e46a0e0bbedf47c, 8c8f4549bec64519aaae94f3c2bf4514; no voters:
I20260811 11:54:26.160794 32518 raft_consensus.cc:2781] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:26.161027 32518 raft_consensus.cc:686] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514 [term 1 LEADER]: Becoming Leader. State: Replica: 8c8f4549bec64519aaae94f3c2bf4514, State: Running, Role: LEADER
I20260811 11:54:26.161402 32518 consensus_queue.cc:237] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659d316c88ed485f8e46a0e0bbedf47c" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37449 } } peers { permanent_uuid: "1ccebf5803594ad7b73b279a35ff1296" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 46155 } } peers { permanent_uuid: "8c8f4549bec64519aaae94f3c2bf4514" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45691 } }
I20260811 11:54:26.165019 31750 catalog_manager.cc:5495] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c8f4549bec64519aaae94f3c2bf4514 (127.3.221.65). New cstate: current_term: 1 leader_uuid: "8c8f4549bec64519aaae94f3c2bf4514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659d316c88ed485f8e46a0e0bbedf47c" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ccebf5803594ad7b73b279a35ff1296" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 46155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c8f4549bec64519aaae94f3c2bf4514" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45691 } health_report { overall_health: HEALTHY } } }
I20260811 11:54:26.194615 32507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:26.190772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45850 (local address 127.3.221.65:45691)
0811 11:54:26.190952 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:26.190956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:26.190972 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:54:26.191128 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:26.191132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:26.191206 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:26.191365 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:54:26.191620 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.192123 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.192979 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.193321 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.193441 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:26.193757 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:54:26.194217 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:54:26.194326 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:26.194403 (+ 77us) server_negotiation.cc:299] Negotiation successful
0811 11:54:26.194467 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 11:54:26.200839 32406 raft_consensus.cc:1261] T c58b88ffe3be4b03a6e451ad658f8e49 P 659d316c88ed485f8e46a0e0bbedf47c [term 1 FOLLOWER]: Refusing update from remote peer 8c8f4549bec64519aaae94f3c2bf4514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:54:26.200846 32181 raft_consensus.cc:1261] T c58b88ffe3be4b03a6e451ad658f8e49 P 1ccebf5803594ad7b73b279a35ff1296 [term 1 FOLLOWER]: Refusing update from remote peer 8c8f4549bec64519aaae94f3c2bf4514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:54:26.201550 32527 consensus_queue.cc:1035] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "659d316c88ed485f8e46a0e0bbedf47c" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 37449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:54:26.201748 32518 consensus_queue.cc:1035] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ccebf5803594ad7b73b279a35ff1296" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 46155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:54:26.213335 32538 mvcc.cc:205] Tried to move back new op lower bound from 7317296194350858240 to 7317296194199814144. Current Snapshot: MvccSnapshot[applied={T|T < 7317296194350858240}]
I20260811 11:54:26.213321 32532 mvcc.cc:205] Tried to move back new op lower bound from 7317296194350858240 to 7317296194199814144. Current Snapshot: MvccSnapshot[applied={T|T < 7317296194350858240}]
I20260811 11:54:26.225046 31748 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:54:26.225306 31748 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:54:26.228137 31748 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4fc973172d164de0b390650483af5d36: Sending DeleteTablet for 3 replicas of tablet c58b88ffe3be4b03a6e451ad658f8e49
I20260811 11:54:26.229347 31942 tablet_service.cc:1514] Processing DeleteTablet for tablet c58b88ffe3be4b03a6e451ad658f8e49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:54:26 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:45842
I20260811 11:54:26.229353 32372 tablet_service.cc:1514] Processing DeleteTablet for tablet c58b88ffe3be4b03a6e451ad658f8e49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:54:26 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:50238
I20260811 11:54:26.229938 32548 tablet_replica.cc:330] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514: stopping tablet replica
I20260811 11:54:26.229979 32145 tablet_service.cc:1514] Processing DeleteTablet for tablet c58b88ffe3be4b03a6e451ad658f8e49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:54:26 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:42118
I20260811 11:54:26.230633 32552 tablet_replica.cc:330] T c58b88ffe3be4b03a6e451ad658f8e49 P 1ccebf5803594ad7b73b279a35ff1296: stopping tablet replica
I20260811 11:54:26.231243 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 31840
I20260811 11:54:26.231272 32552 raft_consensus.cc:2229] T c58b88ffe3be4b03a6e451ad658f8e49 P 1ccebf5803594ad7b73b279a35ff1296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:54:26.231408 32548 raft_consensus.cc:2229] T c58b88ffe3be4b03a6e451ad658f8e49 P 8c8f4549bec64519aaae94f3c2bf4514 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:54:26.231863 32552 raft_consensus.cc:2258] T c58b88ffe3be4b03a6e451ad658f8e49 P 1ccebf5803594ad7b73b279a35ff1296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:54:26.231978 32549 tablet_replica.cc:330] T c58b88ffe3be4b03a6e451ad658f8e49 P 659d316c88ed485f8e46a0e0bbedf47c: stopping tablet replica
I20260811 11:54:26.232292 32549 raft_consensus.cc:2229] T c58b88ffe3be4b03a6e451ad658f8e49 P 659d316c88ed485f8e46a0e0bbedf47c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:54:26.232584 32549 raft_consensus.cc:2258] T c58b88ffe3be4b03a6e451ad658f8e49 P 659d316c88ed485f8e46a0e0bbedf47c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:54:26.235767 32549 ts_tablet_manager.cc:1905] T c58b88ffe3be4b03a6e451ad658f8e49 P 659d316c88ed485f8e46a0e0bbedf47c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:54:26.235769 32552 ts_tablet_manager.cc:1905] T c58b88ffe3be4b03a6e451ad658f8e49 P 1ccebf5803594ad7b73b279a35ff1296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:54:26.238368 32549 ts_tablet_manager.cc:1918] T c58b88ffe3be4b03a6e451ad658f8e49 P 659d316c88ed485f8e46a0e0bbedf47c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:54:26.238472 32549 log.cc:1196] T c58b88ffe3be4b03a6e451ad658f8e49 P 659d316c88ed485f8e46a0e0bbedf47c: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/c58b88ffe3be4b03a6e451ad658f8e49
I20260811 11:54:26.238749 32549 ts_tablet_manager.cc:1939] T c58b88ffe3be4b03a6e451ad658f8e49 P 659d316c88ed485f8e46a0e0bbedf47c: Deleting consensus metadata
I20260811 11:54:26.239068 32552 ts_tablet_manager.cc:1918] T c58b88ffe3be4b03a6e451ad658f8e49 P 1ccebf5803594ad7b73b279a35ff1296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:54:26.239187 32552 log.cc:1196] T c58b88ffe3be4b03a6e451ad658f8e49 P 1ccebf5803594ad7b73b279a35ff1296: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/c58b88ffe3be4b03a6e451ad658f8e49
I20260811 11:54:26.239555 32552 ts_tablet_manager.cc:1939] T c58b88ffe3be4b03a6e451ad658f8e49 P 1ccebf5803594ad7b73b279a35ff1296: Deleting consensus metadata
I20260811 11:54:26.240087 31724 catalog_manager.cc:4841] TS 659d316c88ed485f8e46a0e0bbedf47c (127.3.221.67:37449): tablet c58b88ffe3be4b03a6e451ad658f8e49 (table test-table [id=f1d3ee71853140e38669767dd7bede99]) successfully deleted
I20260811 11:54:26.242063 31725 catalog_manager.cc:4841] TS 1ccebf5803594ad7b73b279a35ff1296 (127.3.221.66:46155): tablet c58b88ffe3be4b03a6e451ad658f8e49 (table test-table [id=f1d3ee71853140e38669767dd7bede99]) successfully deleted
W20260811 11:54:26.248239 31724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.221.65:45691 (error 108)
I20260811 11:54:26.248518 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 32054
I20260811 11:54:26.254870 32499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:26.251834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:45691 (local address 127.0.0.1:45864)
0811 11:54:26.254651 (+ 2817us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:26.254753 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.65:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2706}
W20260811 11:54:26.256340 31724 catalog_manager.cc:4584] TS 8c8f4549bec64519aaae94f3c2bf4514 (127.3.221.65:45691): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c58b88ffe3be4b03a6e451ad658f8e49: Network error: Client connection negotiation failed: client connection to 127.3.221.65:45691: connect: Connection refused (error 111)
I20260811 11:54:26.265000 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 32267
I20260811 11:54:26.288282 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 31698
2026-08-11T11:54:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2671 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-11T11:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:54:26Z Disabled control of system clock
I20260811 11:54:26.327663 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:43105
--webserver_interface=127.3.221.126
--webserver_port=0
--builtin_ntp_servers=127.3.221.84:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:43105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 11:54:26.449847 32560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:26.450160 32560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:26.450198 32560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:26.454403 32560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:54:26.454480 32560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:26.454496 32560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:26.454514 32560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:54:26.454530 32560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:54:26.459896 32560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:43105
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:43105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:26.461104 32560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:26.462626 32560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:26.472455 32573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:26.472455 32568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:26.472455 32570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:26.472824 32560 server_base.cc:1034] running on GCE node
I20260811 11:54:26.473614 32560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:26.474817 32560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:26.476018 32560 hybrid_clock.cc:648] HybridClock initialized: now 1786449266475993 us; error 50 us; skew 500 ppm
I20260811 11:54:26.478680 32560 webserver.cc:460] Webserver started at http://127.3.221.126:33013/ using document root <none> and password file <none>
I20260811 11:54:26.479399 32560 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:26.479494 32560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:26.479861 32560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:26.482632 32560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "3548cffc2e114627bf20706f8c139f5a"
format_stamp: "Formatted at 2026-08-11 11:54:26 on dist-test-slave-bndk"
server_key: "1db90bff589ca8515ae7eb601aaa32fa"
server_key_iv: "4c6e7a8e9df2447a7ab8b36d1283a8b8"
server_key_version: "encryptionkey@0"
I20260811 11:54:26.483951 32560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "3548cffc2e114627bf20706f8c139f5a"
format_stamp: "Formatted at 2026-08-11 11:54:26 on dist-test-slave-bndk"
server_key: "1db90bff589ca8515ae7eb601aaa32fa"
server_key_iv: "4c6e7a8e9df2447a7ab8b36d1283a8b8"
server_key_version: "encryptionkey@0"
I20260811 11:54:26.488883 32560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 11:54:26.492102 32582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:26.493956 32560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260811 11:54:26.494117 32560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "3548cffc2e114627bf20706f8c139f5a"
format_stamp: "Formatted at 2026-08-11 11:54:26 on dist-test-slave-bndk"
server_key: "1db90bff589ca8515ae7eb601aaa32fa"
server_key_iv: "4c6e7a8e9df2447a7ab8b36d1283a8b8"
server_key_version: "encryptionkey@0"
I20260811 11:54:26.494335 32560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:26.512542 32560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:26.513469 32560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:26.513671 32560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:26.521303 32560 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:43105
I20260811 11:54:26.521349 32665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:43105 every 8 connection(s)
I20260811 11:54:26.522359 32560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:54:26.525621 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 32560
I20260811 11:54:26.525774 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:54:26.526074 32671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:26.526105 3957 external_mini_cluster.cc:1428] Setting key 379321d572b6827b70cdc14a308018d0
I20260811 11:54:26.533073 32671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a: Bootstrap starting.
I20260811 11:54:26.535629 32671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:26.535939 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:26.527674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45382 (local address 127.3.221.126:43105)
0811 11:54:26.528428 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:26.528445 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:26.528483 (+ 38us) server_negotiation.cc:407] Connection header received
0811 11:54:26.529135 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:26.529153 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:26.529655 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:26.529986 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:26.530476 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.531493 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.532443 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.532966 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.533133 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:26.534053 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:26.534059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:26.534074 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:26.534503 (+ 429us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:26.534567 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:26.534589 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:26.534681 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:26.534837 (+ 156us) server_negotiation.cc:299] Negotiation successful
0811 11:54:26.535121 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":158,"threads_started":1}
I20260811 11:54:26.536545 32671 log.cc:824] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:26.539077 32671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a: No bootstrap required, opened a new log
I20260811 11:54:26.542976 32671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3548cffc2e114627bf20706f8c139f5a" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 43105 } }
I20260811 11:54:26.543272 32671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:26.543311 32671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3548cffc2e114627bf20706f8c139f5a, State: Initialized, Role: FOLLOWER
I20260811 11:54:26.543787 32671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3548cffc2e114627bf20706f8c139f5a" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 43105 } }
I20260811 11:54:26.543928 32671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:54:26.543993 32671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:54:26.544100 32671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:26.545315 32671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3548cffc2e114627bf20706f8c139f5a" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 43105 } }
I20260811 11:54:26.545680 32671 leader_election.cc:302] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3548cffc2e114627bf20706f8c139f5a; no voters:
I20260811 11:54:26.546025 32671 leader_election.cc:288] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:54:26.546170 32677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:26.546420 32677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a [term 1 LEADER]: Becoming Leader. State: Replica: 3548cffc2e114627bf20706f8c139f5a, State: Running, Role: LEADER
I20260811 11:54:26.546747 32677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3548cffc2e114627bf20706f8c139f5a" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 43105 } }
I20260811 11:54:26.547335 32671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:54:26.548450 32680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3548cffc2e114627bf20706f8c139f5a. Latest consensus state: current_term: 1 leader_uuid: "3548cffc2e114627bf20706f8c139f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3548cffc2e114627bf20706f8c139f5a" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 43105 } } }
I20260811 11:54:26.548596 32680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:26.548869 32678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3548cffc2e114627bf20706f8c139f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3548cffc2e114627bf20706f8c139f5a" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 43105 } } }
I20260811 11:54:26.548970 32678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:26.549110 32691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:54:26.552250 32691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:54:26.559158 32691 catalog_manager.cc:1347] Generated new cluster ID: eaabd4b1e9a54fe5a040f5e05a09752c
I20260811 11:54:26.559257 32691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:54:26.583316 32691 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:54:26.584359 32691 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:54:26.590708 32691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a: Generated new TSK 0
I20260811 11:54:26.591590 32691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:54:26.604691 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:43105
--builtin_ntp_servers=127.3.221.84:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 11:54:26.732230 32711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:26.732540 32711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:26.732582 32711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:26.736956 32711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:26.737049 32711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:26.737108 32711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:54:26.742311 32711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:43105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:26.743525 32711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:26.745000 32711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:26.755378 32718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:26.755427 32720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:26.755488 32711 server_base.cc:1034] running on GCE node
W20260811 11:54:26.755402 32723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:26.756192 32711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:26.756899 32711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:26.758174 32711 hybrid_clock.cc:648] HybridClock initialized: now 1786449266758125 us; error 69 us; skew 500 ppm
I20260811 11:54:26.760282 32711 webserver.cc:460] Webserver started at http://127.3.221.65:42679/ using document root <none> and password file <none>
I20260811 11:54:26.760917 32711 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:26.760994 32711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:26.761233 32711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:26.763505 32711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "d9c0b066543c43338e9fa82d4ba1304b"
format_stamp: "Formatted at 2026-08-11 11:54:26 on dist-test-slave-bndk"
server_key: "bbff6fa7f4447f0719d4c63f2c10054e"
server_key_iv: "20141e561d9978f7fe654bbb08f52e56"
server_key_version: "encryptionkey@0"
I20260811 11:54:26.764102 32711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "d9c0b066543c43338e9fa82d4ba1304b"
format_stamp: "Formatted at 2026-08-11 11:54:26 on dist-test-slave-bndk"
server_key: "bbff6fa7f4447f0719d4c63f2c10054e"
server_key_iv: "20141e561d9978f7fe654bbb08f52e56"
server_key_version: "encryptionkey@0"
I20260811 11:54:26.768180 32711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 11:54:26.771055 32730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:26.772504 32711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 11:54:26.772656 32711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "d9c0b066543c43338e9fa82d4ba1304b"
format_stamp: "Formatted at 2026-08-11 11:54:26 on dist-test-slave-bndk"
server_key: "bbff6fa7f4447f0719d4c63f2c10054e"
server_key_iv: "20141e561d9978f7fe654bbb08f52e56"
server_key_version: "encryptionkey@0"
I20260811 11:54:26.772781 32711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:26.788686 32711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:26.789485 32711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:26.789677 32711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:26.790303 32711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:54:26.791343 32711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:54:26.791402 32711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:26.791460 32711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:54:26.791486 32711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:26.800814 32711 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:38419
I20260811 11:54:26.800848 432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:38419 every 8 connection(s)
I20260811 11:54:26.801841 32711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:54:26.803342 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 32711
I20260811 11:54:26.803443 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:54:26.803728 3957 external_mini_cluster.cc:1428] Setting key 91d5458dde6e552d33feec15063a2f64
I20260811 11:54:26.805847 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:43105
--builtin_ntp_servers=127.3.221.84:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 11:54:26.812125 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:26.803889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:44555 (local address 127.3.221.126:43105)
0811 11:54:26.804063 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:26.804068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:26.805358 (+ 1290us) server_negotiation.cc:407] Connection header received
0811 11:54:26.806829 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:26.806832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:26.806894 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:26.806984 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:26.807942 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.808717 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.810005 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.810224 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.810317 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:26.811600 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:26.811604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:26.811606 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:26.811620 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:26.811640 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:26.811648 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:26.811711 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:26.811860 (+ 149us) server_negotiation.cc:299] Negotiation successful
0811 11:54:26.811964 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 11:54:26.813025 438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:26.804407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:43105 (local address 127.3.221.65:44555)
0811 11:54:26.805151 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:26.805205 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:26.806606 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:26.807157 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:26.807164 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:26.807269 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:54:26.807795 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:26.807809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.808876 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.808880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:26.809867 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:26.809879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:26.810378 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:26.810383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:26.810556 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:26.811290 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:54:26.811292 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:54:26.811338 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:54:26.811425 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:54:26.811723 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:54:26.811733 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:54:26.811739 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 11:54:26.811852 (+ 113us) client_negotiation.cc:240] Negotiation successful
0811 11:54:26.812164 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":120,"threads_started":1}
I20260811 11:54:26.814373 433 heartbeater.cc:347] Connected to a master server at 127.3.221.126:43105
I20260811 11:54:26.814617 433 heartbeater.cc:464] Registering TS with master...
I20260811 11:54:26.815178 433 heartbeater.cc:511] Master 127.3.221.126:43105 requested a full tablet report, sending...
I20260811 11:54:26.817134 32605 ts_manager.cc:194] Registered new tserver with Master: d9c0b066543c43338e9fa82d4ba1304b (127.3.221.65:38419)
I20260811 11:54:26.818827 32605 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.221.65:44555
W20260811 11:54:26.928849 439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:26.929147 439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:26.929186 439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:26.933384 439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:26.933450 439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:26.933509 439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:54:26.938701 439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:43105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:26.939880 439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:26.941308 439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:26.951938 447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:26.952119 439 server_base.cc:1034] running on GCE node
W20260811 11:54:26.951969 450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:26.952008 446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:26.952731 439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:26.953459 439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:26.954643 439 hybrid_clock.cc:648] HybridClock initialized: now 1786449266954622 us; error 34 us; skew 500 ppm
I20260811 11:54:26.956707 439 webserver.cc:460] Webserver started at http://127.3.221.66:35455/ using document root <none> and password file <none>
I20260811 11:54:26.957405 439 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:26.957486 439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:26.957700 439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:26.959935 439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "da87afbbb6924811b7ba72cf74679bed"
format_stamp: "Formatted at 2026-08-11 11:54:26 on dist-test-slave-bndk"
server_key: "091d334fc61a3863c35577be8e5aaad1"
server_key_iv: "c564a4a0ab02601130510fd89e1f5178"
server_key_version: "encryptionkey@0"
I20260811 11:54:26.960508 439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "da87afbbb6924811b7ba72cf74679bed"
format_stamp: "Formatted at 2026-08-11 11:54:26 on dist-test-slave-bndk"
server_key: "091d334fc61a3863c35577be8e5aaad1"
server_key_iv: "c564a4a0ab02601130510fd89e1f5178"
server_key_version: "encryptionkey@0"
I20260811 11:54:26.964581 439 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 11:54:26.967367 461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:26.968801 439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260811 11:54:26.968945 439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "da87afbbb6924811b7ba72cf74679bed"
format_stamp: "Formatted at 2026-08-11 11:54:26 on dist-test-slave-bndk"
server_key: "091d334fc61a3863c35577be8e5aaad1"
server_key_iv: "c564a4a0ab02601130510fd89e1f5178"
server_key_version: "encryptionkey@0"
I20260811 11:54:26.969087 439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:26.984877 439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:26.985713 439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:26.985913 439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:26.986573 439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:54:26.987620 439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:54:26.987671 439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:26.987712 439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:54:26.987741 439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:26.996838 439 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:39055
I20260811 11:54:26.996867 635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:39055 every 8 connection(s)
I20260811 11:54:26.997835 439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:54:27.004311 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 439
I20260811 11:54:27.004453 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:54:27.004743 3957 external_mini_cluster.cc:1428] Setting key 23371965ec301249e97f5d94a47080fb
I20260811 11:54:27.006913 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:43105
--builtin_ntp_servers=127.3.221.84:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 11:54:27.008287 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:26.999828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:47563 (local address 127.3.221.126:43105)
0811 11:54:26.999973 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:26.999977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:27.001729 (+ 1752us) server_negotiation.cc:407] Connection header received
0811 11:54:27.003232 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:27.003235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:27.003297 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:27.003375 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:27.004353 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.004945 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.006328 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.006551 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.006640 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:27.007811 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:27.007815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:27.007818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:27.007832 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:27.007852 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:27.007860 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:27.007927 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:27.008071 (+ 144us) server_negotiation.cc:299] Negotiation successful
0811 11:54:27.008139 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 11:54:27.009524 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:27.000307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:43105 (local address 127.3.221.66:47563)
0811 11:54:27.001526 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:27.001589 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:27.002992 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:27.003581 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:27.003588 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:27.003683 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:54:27.004204 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:27.004218 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.005078 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.005081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:27.006211 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:27.006223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.006701 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.006705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:27.006871 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:27.007522 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:54:27.007524 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:54:27.007581 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:54:27.007655 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:54:27.007930 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:54:27.007940 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:54:27.007947 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 11:54:27.008069 (+ 122us) client_negotiation.cc:240] Negotiation successful
0811 11:54:27.008390 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":129,"threads_started":1}
I20260811 11:54:27.010859 636 heartbeater.cc:347] Connected to a master server at 127.3.221.126:43105
I20260811 11:54:27.011106 636 heartbeater.cc:464] Registering TS with master...
I20260811 11:54:27.011701 636 heartbeater.cc:511] Master 127.3.221.126:43105 requested a full tablet report, sending...
I20260811 11:54:27.012938 32605 ts_manager.cc:194] Registered new tserver with Master: da87afbbb6924811b7ba72cf74679bed (127.3.221.66:39055)
I20260811 11:54:27.013758 32605 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.221.66:47563
W20260811 11:54:27.130496 642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:27.130805 642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:27.130847 642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:27.135041 642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:27.135104 642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:27.135164 642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:54:27.140175 642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:43105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:27.141360 642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:27.142755 642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:27.153079 656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:27.153134 642 server_base.cc:1034] running on GCE node
W20260811 11:54:27.153081 651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:27.153084 652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:27.153928 642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:27.154596 642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:27.155791 642 hybrid_clock.cc:648] HybridClock initialized: now 1786449267155763 us; error 44 us; skew 500 ppm
I20260811 11:54:27.157943 642 webserver.cc:460] Webserver started at http://127.3.221.67:40831/ using document root <none> and password file <none>
I20260811 11:54:27.158584 642 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:27.158661 642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:27.158881 642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:27.161106 642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "928af9872c79451386174bfcfe02a282"
format_stamp: "Formatted at 2026-08-11 11:54:27 on dist-test-slave-bndk"
server_key: "41beb55e45ab94060d39e6d046859d4b"
server_key_iv: "7a75f2e5cc8224bc29db7d2dd4258e08"
server_key_version: "encryptionkey@0"
I20260811 11:54:27.161760 642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "928af9872c79451386174bfcfe02a282"
format_stamp: "Formatted at 2026-08-11 11:54:27 on dist-test-slave-bndk"
server_key: "41beb55e45ab94060d39e6d046859d4b"
server_key_iv: "7a75f2e5cc8224bc29db7d2dd4258e08"
server_key_version: "encryptionkey@0"
I20260811 11:54:27.165789 642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 11:54:27.168923 664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:27.170701 642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:54:27.170859 642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "928af9872c79451386174bfcfe02a282"
format_stamp: "Formatted at 2026-08-11 11:54:27 on dist-test-slave-bndk"
server_key: "41beb55e45ab94060d39e6d046859d4b"
server_key_iv: "7a75f2e5cc8224bc29db7d2dd4258e08"
server_key_version: "encryptionkey@0"
I20260811 11:54:27.171021 642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:27.189023 642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:27.189872 642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:27.190047 642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:27.190657 642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:54:27.191742 642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:54:27.191802 642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:27.191845 642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:54:27.191870 642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:27.203497 642 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:39355
I20260811 11:54:27.203533 834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:39355 every 8 connection(s)
I20260811 11:54:27.204811 642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:54:27.210050 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 642
I20260811 11:54:27.210187 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:54:27.210500 3957 external_mini_cluster.cc:1428] Setting key 6b949f746f81be2c2713ccfa6cafb761
I20260811 11:54:27.216321 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:27.206980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:35075 (local address 127.3.221.126:43105)
0811 11:54:27.207118 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:27.207121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:27.208295 (+ 1174us) server_negotiation.cc:407] Connection header received
0811 11:54:27.209887 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:27.209890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:27.209946 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:27.210024 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:27.211319 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.211989 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.213600 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.213926 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.214077 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:27.215708 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:27.215712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:27.215714 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:27.215729 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:27.215759 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:27.215769 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:27.215849 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:27.216112 (+ 263us) server_negotiation.cc:299] Negotiation successful
0811 11:54:27.216188 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 11:54:27.217350 842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:27.207356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:43105 (local address 127.3.221.67:35075)
0811 11:54:27.208079 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:27.208125 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:27.209629 (+ 1504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:27.210249 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:27.210258 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:27.210377 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:54:27.211064 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:27.211085 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.212142 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.212146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:27.213345 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:27.213357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.214148 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.214152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:27.214345 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:27.215236 (+ 891us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:54:27.215238 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:54:27.215319 (+ 81us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:54:27.215448 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:54:27.215910 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:54:27.215921 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:54:27.215927 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 11:54:27.216080 (+ 153us) client_negotiation.cc:240] Negotiation successful
0811 11:54:27.216383 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":128,"threads_started":1}
I20260811 11:54:27.218532 839 heartbeater.cc:347] Connected to a master server at 127.3.221.126:43105
I20260811 11:54:27.218770 839 heartbeater.cc:464] Registering TS with master...
I20260811 11:54:27.219314 839 heartbeater.cc:511] Master 127.3.221.126:43105 requested a full tablet report, sending...
I20260811 11:54:27.220378 32605 ts_manager.cc:194] Registered new tserver with Master: 928af9872c79451386174bfcfe02a282 (127.3.221.67:39355)
I20260811 11:54:27.220997 32605 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.221.67:35075
I20260811 11:54:27.225291 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:54:27.231468 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:27.227348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35886 (local address 127.3.221.126:43105)
0811 11:54:27.227475 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:27.227479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:27.227587 (+ 108us) server_negotiation.cc:407] Connection header received
0811 11:54:27.227795 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:27.227797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:27.227854 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:27.227944 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:27.228191 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.228816 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.229835 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.230062 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.230152 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:27.230978 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:27.230981 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:27.230983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:27.230996 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:27.231018 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:27.231026 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:27.231088 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:27.231188 (+ 100us) server_negotiation.cc:299] Negotiation successful
0811 11:54:27.231264 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 11:54:27.234856 32605 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:54:27.238140 32605 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:54:27.255584 862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:27.250705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53704 (local address 127.3.221.65:38419)
0811 11:54:27.251145 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:27.251157 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:27.251179 (+ 22us) server_negotiation.cc:407] Connection header received
0811 11:54:27.251271 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:27.251277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:27.251470 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:27.251629 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:27.251831 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.252892 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.254242 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.254849 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.254957 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:27.255192 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:27.255305 (+ 113us) server_negotiation.cc:299] Negotiation successful
0811 11:54:27.255416 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":121,"threads_started":1}
I20260811 11:54:27.255579 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:27.250134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:38419 (local address 127.0.0.1:53704)
0811 11:54:27.250378 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:27.250391 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:27.250479 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:27.251619 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:27.251622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:27.251641 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:27.251728 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:27.251735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.252999 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.253002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:27.254106 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:27.254117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.254986 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.254989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:27.255101 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:27.255140 (+ 39us) client_negotiation.cc:769] Sending connection context
0811 11:54:27.255263 (+ 123us) client_negotiation.cc:240] Negotiation successful
0811 11:54:27.255410 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":48,"thread_start_us":102,"threads_started":1}
I20260811 11:54:27.255894 859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:27.249191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:39055 (local address 127.0.0.1:55822)
0811 11:54:27.249804 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:27.249859 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:27.250024 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:27.251087 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:27.251093 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:27.251140 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:27.251361 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:27.251370 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.252799 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.252803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:27.254101 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:27.254109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.255421 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.255425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:27.255548 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:27.255588 (+ 40us) client_negotiation.cc:769] Sending connection context
0811 11:54:27.255660 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 11:54:27.255744 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":115,"spinlock_wait_cycles":5888,"thread_start_us":99,"threads_started":1}
I20260811 11:54:27.256356 856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:27.249275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:39355 (local address 127.0.0.1:32830)
0811 11:54:27.250197 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:27.250212 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:27.250317 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:27.251087 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:27.251094 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:27.251168 (+ 74us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:27.251339 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:27.251347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.252484 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.252487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:27.254033 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:27.254047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.255427 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.255430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:27.255552 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:27.255600 (+ 48us) client_negotiation.cc:769] Sending connection context
0811 11:54:27.256033 (+ 433us) client_negotiation.cc:240] Negotiation successful
0811 11:54:27.256115 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"mutex_wait_us":158,"spinlock_wait_cycles":85888,"thread_start_us":148,"threads_started":1}
I20260811 11:54:27.256578 857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:27.249314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55822 (local address 127.3.221.66:39055)
0811 11:54:27.250352 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:27.250363 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:27.250392 (+ 29us) server_negotiation.cc:407] Connection header received
0811 11:54:27.250484 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:27.250489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:27.250726 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:27.250892 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:27.251499 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.252669 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.254225 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.255053 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.255162 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:27.255348 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:27.256210 (+ 862us) server_negotiation.cc:299] Negotiation successful
0811 11:54:27.256419 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":918,"thread_start_us":134,"threads_started":1}
I20260811 11:54:27.256664 860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:27.249759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32830 (local address 127.3.221.67:39355)
0811 11:54:27.250131 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:27.250144 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:27.250424 (+ 280us) server_negotiation.cc:407] Connection header received
0811 11:54:27.250511 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:27.250517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:27.250726 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:27.250893 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:27.251467 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.252363 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.254199 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.255053 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.255162 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:27.255350 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:27.255850 (+ 500us) server_negotiation.cc:299] Negotiation successful
0811 11:54:27.256507 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":161,"threads_started":1}
I20260811 11:54:27.259330 331 tablet_service.cc:1467] Processing CreateTablet for tablet 3bfd584261a74bbfad2727c0ddab89d1 (DEFAULT_TABLE table=test-table [id=9c0ed34565f34fa59bcb00dd5bf49831]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:54:27.259716 533 tablet_service.cc:1467] Processing CreateTablet for tablet 3bfd584261a74bbfad2727c0ddab89d1 (DEFAULT_TABLE table=test-table [id=9c0ed34565f34fa59bcb00dd5bf49831]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:54:27.260455 331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bfd584261a74bbfad2727c0ddab89d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:27.260350 734 tablet_service.cc:1467] Processing CreateTablet for tablet 3bfd584261a74bbfad2727c0ddab89d1 (DEFAULT_TABLE table=test-table [id=9c0ed34565f34fa59bcb00dd5bf49831]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:54:27.260811 533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bfd584261a74bbfad2727c0ddab89d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:27.261408 734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bfd584261a74bbfad2727c0ddab89d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:27.268383 863 tablet_bootstrap.cc:492] T 3bfd584261a74bbfad2727c0ddab89d1 P d9c0b066543c43338e9fa82d4ba1304b: Bootstrap starting.
I20260811 11:54:27.269048 865 tablet_bootstrap.cc:492] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed: Bootstrap starting.
I20260811 11:54:27.270047 866 tablet_bootstrap.cc:492] T 3bfd584261a74bbfad2727c0ddab89d1 P 928af9872c79451386174bfcfe02a282: Bootstrap starting.
I20260811 11:54:27.271337 865 tablet_bootstrap.cc:654] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:27.271328 863 tablet_bootstrap.cc:654] T 3bfd584261a74bbfad2727c0ddab89d1 P d9c0b066543c43338e9fa82d4ba1304b: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:27.272080 866 tablet_bootstrap.cc:654] T 3bfd584261a74bbfad2727c0ddab89d1 P 928af9872c79451386174bfcfe02a282: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:27.272357 865 log.cc:824] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:27.272357 863 log.cc:824] T 3bfd584261a74bbfad2727c0ddab89d1 P d9c0b066543c43338e9fa82d4ba1304b: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:27.272872 866 log.cc:824] T 3bfd584261a74bbfad2727c0ddab89d1 P 928af9872c79451386174bfcfe02a282: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:27.274765 863 tablet_bootstrap.cc:492] T 3bfd584261a74bbfad2727c0ddab89d1 P d9c0b066543c43338e9fa82d4ba1304b: No bootstrap required, opened a new log
I20260811 11:54:27.274765 865 tablet_bootstrap.cc:492] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed: No bootstrap required, opened a new log
I20260811 11:54:27.274999 865 ts_tablet_manager.cc:1397] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 11:54:27.274999 863 ts_tablet_manager.cc:1397] T 3bfd584261a74bbfad2727c0ddab89d1 P d9c0b066543c43338e9fa82d4ba1304b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260811 11:54:27.279003 866 tablet_bootstrap.cc:492] T 3bfd584261a74bbfad2727c0ddab89d1 P 928af9872c79451386174bfcfe02a282: No bootstrap required, opened a new log
I20260811 11:54:27.279218 866 ts_tablet_manager.cc:1397] T 3bfd584261a74bbfad2727c0ddab89d1 P 928af9872c79451386174bfcfe02a282: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20260811 11:54:27.279528 865 raft_consensus.cc:348] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "928af9872c79451386174bfcfe02a282" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 39355 } } peers { permanent_uuid: "da87afbbb6924811b7ba72cf74679bed" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 39055 } } peers { permanent_uuid: "d9c0b066543c43338e9fa82d4ba1304b" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 38419 } }
I20260811 11:54:27.279528 863 raft_consensus.cc:348] T 3bfd584261a74bbfad2727c0ddab89d1 P d9c0b066543c43338e9fa82d4ba1304b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "928af9872c79451386174bfcfe02a282" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 39355 } } peers { permanent_uuid: "da87afbbb6924811b7ba72cf74679bed" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 39055 } } peers { permanent_uuid: "d9c0b066543c43338e9fa82d4ba1304b" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 38419 } }
I20260811 11:54:27.279834 865 raft_consensus.cc:374] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:27.279834 863 raft_consensus.cc:374] T 3bfd584261a74bbfad2727c0ddab89d1 P d9c0b066543c43338e9fa82d4ba1304b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:27.279896 865 raft_consensus.cc:729] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da87afbbb6924811b7ba72cf74679bed, State: Initialized, Role: FOLLOWER
I20260811 11:54:27.279908 863 raft_consensus.cc:729] T 3bfd584261a74bbfad2727c0ddab89d1 P d9c0b066543c43338e9fa82d4ba1304b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c0b066543c43338e9fa82d4ba1304b, State: Initialized, Role: FOLLOWER
I20260811 11:54:27.280396 865 consensus_queue.cc:260] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "928af9872c79451386174bfcfe02a282" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 39355 } } peers { permanent_uuid: "da87afbbb6924811b7ba72cf74679bed" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 39055 } } peers { permanent_uuid: "d9c0b066543c43338e9fa82d4ba1304b" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 38419 } }
I20260811 11:54:27.280396 863 consensus_queue.cc:260] T 3bfd584261a74bbfad2727c0ddab89d1 P d9c0b066543c43338e9fa82d4ba1304b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "928af9872c79451386174bfcfe02a282" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 39355 } } peers { permanent_uuid: "da87afbbb6924811b7ba72cf74679bed" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 39055 } } peers { permanent_uuid: "d9c0b066543c43338e9fa82d4ba1304b" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 38419 } }
I20260811 11:54:27.281648 863 ts_tablet_manager.cc:1428] T 3bfd584261a74bbfad2727c0ddab89d1 P d9c0b066543c43338e9fa82d4ba1304b: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260811 11:54:27.281648 865 ts_tablet_manager.cc:1428] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260811 11:54:27.282060 636 heartbeater.cc:503] Master 127.3.221.126:43105 was elected leader, sending a full tablet report...
I20260811 11:54:27.282279 433 heartbeater.cc:503] Master 127.3.221.126:43105 was elected leader, sending a full tablet report...
I20260811 11:54:27.284024 866 raft_consensus.cc:348] T 3bfd584261a74bbfad2727c0ddab89d1 P 928af9872c79451386174bfcfe02a282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "928af9872c79451386174bfcfe02a282" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 39355 } } peers { permanent_uuid: "da87afbbb6924811b7ba72cf74679bed" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 39055 } } peers { permanent_uuid: "d9c0b066543c43338e9fa82d4ba1304b" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 38419 } }
I20260811 11:54:27.284344 866 raft_consensus.cc:374] T 3bfd584261a74bbfad2727c0ddab89d1 P 928af9872c79451386174bfcfe02a282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:27.284399 866 raft_consensus.cc:729] T 3bfd584261a74bbfad2727c0ddab89d1 P 928af9872c79451386174bfcfe02a282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 928af9872c79451386174bfcfe02a282, State: Initialized, Role: FOLLOWER
I20260811 11:54:27.284960 866 consensus_queue.cc:260] T 3bfd584261a74bbfad2727c0ddab89d1 P 928af9872c79451386174bfcfe02a282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "928af9872c79451386174bfcfe02a282" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 39355 } } peers { permanent_uuid: "da87afbbb6924811b7ba72cf74679bed" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 39055 } } peers { permanent_uuid: "d9c0b066543c43338e9fa82d4ba1304b" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 38419 } }
I20260811 11:54:27.286165 839 heartbeater.cc:503] Master 127.3.221.126:43105 was elected leader, sending a full tablet report...
I20260811 11:54:27.286348 866 ts_tablet_manager.cc:1428] T 3bfd584261a74bbfad2727c0ddab89d1 P 928af9872c79451386174bfcfe02a282: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260811 11:54:27.303076 434 tablet.cc:2369] T 3bfd584261a74bbfad2727c0ddab89d1 P d9c0b066543c43338e9fa82d4ba1304b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:54:27.372141 874 raft_consensus.cc:482] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:54:27.372509 874 raft_consensus.cc:504] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "928af9872c79451386174bfcfe02a282" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 39355 } } peers { permanent_uuid: "da87afbbb6924811b7ba72cf74679bed" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 39055 } } peers { permanent_uuid: "d9c0b066543c43338e9fa82d4ba1304b" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 38419 } }
I20260811 11:54:27.374374 874 leader_election.cc:288] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 928af9872c79451386174bfcfe02a282 (127.3.221.67:39355), d9c0b066543c43338e9fa82d4ba1304b (127.3.221.65:38419)
I20260811 11:54:27.378132 860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:27.374806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:35665 (local address 127.3.221.67:39355)
0811 11:54:27.374946 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:27.374950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:27.374966 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:54:27.375022 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:27.375024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:27.375075 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:27.375146 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:27.375419 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.375945 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.376999 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.377546 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.377662 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:27.377720 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:27.377932 (+ 212us) server_negotiation.cc:299] Negotiation successful
0811 11:54:27.378008 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 11:54:27.378134 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:27.374564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:39355 (local address 127.3.221.66:35665)
0811 11:54:27.374707 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:27.374724 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:27.374833 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:27.375172 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:27.375175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:27.375194 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:27.375312 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:27.375318 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.376051 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.376055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:27.376869 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:27.376878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.377662 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.377666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:27.377781 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:27.377804 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 11:54:27.377899 (+ 95us) client_negotiation.cc:240] Negotiation successful
0811 11:54:27.377983 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260811 11:54:27.378575 862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:27.374902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:53491 (local address 127.3.221.65:38419)
0811 11:54:27.375046 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:27.375051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:27.375276 (+ 225us) server_negotiation.cc:407] Connection header received
0811 11:54:27.375414 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:27.375417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:27.375463 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:27.375527 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:27.375808 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.376358 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.377561 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.377990 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.378096 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:27.378144 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:27.378375 (+ 231us) server_negotiation.cc:299] Negotiation successful
0811 11:54:27.378453 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 11:54:27.378926 879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:27.374794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:38419 (local address 127.3.221.66:53491)
0811 11:54:27.375180 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:27.375193 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:27.375297 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:27.375544 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:27.375547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:27.375563 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:27.375699 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:27.375706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.376460 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.376463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:27.377443 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:27.377453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.378116 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.378119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:27.378211 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:27.378237 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 11:54:27.378729 (+ 492us) client_negotiation.cc:240] Negotiation successful
0811 11:54:27.378808 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":158,"threads_started":1}
I20260811 11:54:27.378877 763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bfd584261a74bbfad2727c0ddab89d1" candidate_uuid: "da87afbbb6924811b7ba72cf74679bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "928af9872c79451386174bfcfe02a282" is_pre_election: true
I20260811 11:54:27.379216 763 raft_consensus.cc:2454] T 3bfd584261a74bbfad2727c0ddab89d1 P 928af9872c79451386174bfcfe02a282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da87afbbb6924811b7ba72cf74679bed in term 0.
I20260811 11:54:27.379868 362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bfd584261a74bbfad2727c0ddab89d1" candidate_uuid: "da87afbbb6924811b7ba72cf74679bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9c0b066543c43338e9fa82d4ba1304b" is_pre_election: true
I20260811 11:54:27.380143 465 leader_election.cc:302] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 928af9872c79451386174bfcfe02a282, da87afbbb6924811b7ba72cf74679bed; no voters:
I20260811 11:54:27.380187 362 raft_consensus.cc:2454] T 3bfd584261a74bbfad2727c0ddab89d1 P d9c0b066543c43338e9fa82d4ba1304b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da87afbbb6924811b7ba72cf74679bed in term 0.
I20260811 11:54:27.380486 874 raft_consensus.cc:2781] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:54:27.380599 874 raft_consensus.cc:482] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:54:27.380657 874 raft_consensus.cc:3037] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:27.381834 874 raft_consensus.cc:504] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "928af9872c79451386174bfcfe02a282" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 39355 } } peers { permanent_uuid: "da87afbbb6924811b7ba72cf74679bed" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 39055 } } peers { permanent_uuid: "d9c0b066543c43338e9fa82d4ba1304b" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 38419 } }
I20260811 11:54:27.382202 874 leader_election.cc:288] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed [CANDIDATE]: Term 1 election: Requested vote from peers 928af9872c79451386174bfcfe02a282 (127.3.221.67:39355), d9c0b066543c43338e9fa82d4ba1304b (127.3.221.65:38419)
I20260811 11:54:27.382671 763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bfd584261a74bbfad2727c0ddab89d1" candidate_uuid: "da87afbbb6924811b7ba72cf74679bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "928af9872c79451386174bfcfe02a282"
I20260811 11:54:27.382670 362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bfd584261a74bbfad2727c0ddab89d1" candidate_uuid: "da87afbbb6924811b7ba72cf74679bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9c0b066543c43338e9fa82d4ba1304b"
I20260811 11:54:27.382835 763 raft_consensus.cc:3037] T 3bfd584261a74bbfad2727c0ddab89d1 P 928af9872c79451386174bfcfe02a282 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:27.382835 362 raft_consensus.cc:3037] T 3bfd584261a74bbfad2727c0ddab89d1 P d9c0b066543c43338e9fa82d4ba1304b [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:27.384207 763 raft_consensus.cc:2454] T 3bfd584261a74bbfad2727c0ddab89d1 P 928af9872c79451386174bfcfe02a282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da87afbbb6924811b7ba72cf74679bed in term 1.
I20260811 11:54:27.384207 362 raft_consensus.cc:2454] T 3bfd584261a74bbfad2727c0ddab89d1 P d9c0b066543c43338e9fa82d4ba1304b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da87afbbb6924811b7ba72cf74679bed in term 1.
I20260811 11:54:27.384694 465 leader_election.cc:302] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 928af9872c79451386174bfcfe02a282, da87afbbb6924811b7ba72cf74679bed; no voters:
I20260811 11:54:27.384909 874 raft_consensus.cc:2781] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:27.385147 874 raft_consensus.cc:686] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed [term 1 LEADER]: Becoming Leader. State: Replica: da87afbbb6924811b7ba72cf74679bed, State: Running, Role: LEADER
I20260811 11:54:27.385542 874 consensus_queue.cc:237] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "928af9872c79451386174bfcfe02a282" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 39355 } } peers { permanent_uuid: "da87afbbb6924811b7ba72cf74679bed" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 39055 } } peers { permanent_uuid: "d9c0b066543c43338e9fa82d4ba1304b" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 38419 } }
I20260811 11:54:27.389418 32605 catalog_manager.cc:5495] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed reported cstate change: term changed from 0 to 1, leader changed from <none> to da87afbbb6924811b7ba72cf74679bed (127.3.221.66). New cstate: current_term: 1 leader_uuid: "da87afbbb6924811b7ba72cf74679bed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "928af9872c79451386174bfcfe02a282" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 39355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da87afbbb6924811b7ba72cf74679bed" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 39055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9c0b066543c43338e9fa82d4ba1304b" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 38419 } health_report { overall_health: UNKNOWN } } }
I20260811 11:54:27.413306 857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:27.409753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55830 (local address 127.3.221.66:39055)
0811 11:54:27.409927 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:27.409932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:27.409947 (+ 15us) server_negotiation.cc:407] Connection header received
0811 11:54:27.410032 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:27.410034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:27.410091 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:27.410187 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:54:27.410424 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.410924 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.411780 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:27.411994 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:27.412083 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:27.412383 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:54:27.412838 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:54:27.412949 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:27.413018 (+ 69us) server_negotiation.cc:299] Negotiation successful
0811 11:54:27.413078 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 11:54:27.419472 763 raft_consensus.cc:1261] T 3bfd584261a74bbfad2727c0ddab89d1 P 928af9872c79451386174bfcfe02a282 [term 1 FOLLOWER]: Refusing update from remote peer da87afbbb6924811b7ba72cf74679bed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:54:27.419472 362 raft_consensus.cc:1261] T 3bfd584261a74bbfad2727c0ddab89d1 P d9c0b066543c43338e9fa82d4ba1304b [term 1 FOLLOWER]: Refusing update from remote peer da87afbbb6924811b7ba72cf74679bed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:54:27.420264 880 consensus_queue.cc:1035] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "928af9872c79451386174bfcfe02a282" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 39355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:54:27.420512 874 consensus_queue.cc:1035] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9c0b066543c43338e9fa82d4ba1304b" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 38419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:54:27.431174 888 mvcc.cc:205] Tried to move back new op lower bound from 7317296199342460928 to 7317296199214120960. Current Snapshot: MvccSnapshot[applied={T|T < 7317296199342460928}]
I20260811 11:54:27.443069 32605 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:54:27.443295 32605 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:54:27.446100 32605 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3548cffc2e114627bf20706f8c139f5a: Sending DeleteTablet for 3 replicas of tablet 3bfd584261a74bbfad2727c0ddab89d1
I20260811 11:54:27.447005 533 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bfd584261a74bbfad2727c0ddab89d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:54:27 UTC) from {username='slave'} at 127.0.0.1:55822
I20260811 11:54:27.447224 331 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bfd584261a74bbfad2727c0ddab89d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:54:27 UTC) from {username='slave'} at 127.0.0.1:53704
I20260811 11:54:27.447647 734 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bfd584261a74bbfad2727c0ddab89d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:54:27 UTC) from {username='slave'} at 127.0.0.1:32830
I20260811 11:54:27.447736 904 tablet_replica.cc:330] T 3bfd584261a74bbfad2727c0ddab89d1 P d9c0b066543c43338e9fa82d4ba1304b: stopping tablet replica
I20260811 11:54:27.448163 904 raft_consensus.cc:2229] T 3bfd584261a74bbfad2727c0ddab89d1 P d9c0b066543c43338e9fa82d4ba1304b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:54:27.448231 905 tablet_replica.cc:330] T 3bfd584261a74bbfad2727c0ddab89d1 P 928af9872c79451386174bfcfe02a282: stopping tablet replica
I20260811 11:54:27.448308 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 32711
I20260811 11:54:27.458098 905 raft_consensus.cc:2229] T 3bfd584261a74bbfad2727c0ddab89d1 P 928af9872c79451386174bfcfe02a282 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:54:27.458436 905 raft_consensus.cc:2258] T 3bfd584261a74bbfad2727c0ddab89d1 P 928af9872c79451386174bfcfe02a282 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:54:27.458809 903 tablet_replica.cc:330] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed: stopping tablet replica
I20260811 11:54:27.459100 903 raft_consensus.cc:2229] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:54:27.459484 903 raft_consensus.cc:2258] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:54:27.461125 905 ts_tablet_manager.cc:1905] T 3bfd584261a74bbfad2727c0ddab89d1 P 928af9872c79451386174bfcfe02a282: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:54:27.463534 905 ts_tablet_manager.cc:1918] T 3bfd584261a74bbfad2727c0ddab89d1 P 928af9872c79451386174bfcfe02a282: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:54:27.463642 905 log.cc:1196] T 3bfd584261a74bbfad2727c0ddab89d1 P 928af9872c79451386174bfcfe02a282: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/3bfd584261a74bbfad2727c0ddab89d1
I20260811 11:54:27.463953 905 ts_tablet_manager.cc:1939] T 3bfd584261a74bbfad2727c0ddab89d1 P 928af9872c79451386174bfcfe02a282: Deleting consensus metadata
I20260811 11:54:27.465493 32585 catalog_manager.cc:4841] TS 928af9872c79451386174bfcfe02a282 (127.3.221.67:39355): tablet 3bfd584261a74bbfad2727c0ddab89d1 (table test-table [id=9c0ed34565f34fa59bcb00dd5bf49831]) successfully deleted
I20260811 11:54:27.465557 903 ts_tablet_manager.cc:1905] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260811 11:54:27.468007 32584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.221.65:38419 (error 108)
I20260811 11:54:27.468073 903 ts_tablet_manager.cc:1918] T 3bfd584261a74bbfad2727c0ddab89d1 P da87afbbb6924811b7ba72cf74679bed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:54:27.468286 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 439
I20260811 11:54:27.495033 856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:27.494519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:38419 (local address 127.0.0.1:53706)
0811 11:54:27.494808 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:27.494912 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.65:38419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
W20260811 11:54:27.495294 32584 catalog_manager.cc:4584] TS d9c0b066543c43338e9fa82d4ba1304b (127.3.221.65:38419): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3bfd584261a74bbfad2727c0ddab89d1: Network error: Client connection negotiation failed: client connection to 127.3.221.65:38419: connect: Connection refused (error 111)
I20260811 11:54:27.504877 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 642
I20260811 11:54:27.511292 856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:27.505052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:39055 (local address 127.0.0.1:55840)
0811 11:54:27.510950 (+ 5898us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:27.511120 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.66:39055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5621}
I20260811 11:54:27.517350 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 32560
2026-08-11T11:54:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1228 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:54:27 dist-test-slave-bndk krb5kdc[912](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:54:27 dist-test-slave-bndk krb5kdc[912](info): set up 2 sockets
Aug 11 11:54:27 dist-test-slave-bndk krb5kdc[912](info): commencing operation
krb5kdc: starting...
W20260811 11:54:28.720366 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:54:28 dist-test-slave-bndk krb5kdc[912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:54:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:54:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:54:28.861742 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:36279
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:36279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:28.986568 942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:28.986892 942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:28.986936 942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:28.991364 942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:54:28.991449 942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:28.991470 942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:28.991487 942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:54:28.991501 942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:54:28.997166 942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:36279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:36279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:28.998538 942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:28.999715 942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:29.009260 955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:29.009303 942 server_base.cc:1034] running on GCE node
W20260811 11:54:29.009240 953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:29.009236 957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:29.010104 942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:29.011340 942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:29.012539 942 hybrid_clock.cc:648] HybridClock initialized: now 1786449269012517 us; error 39 us; skew 500 ppm
Aug 11 11:54:29 dist-test-slave-bndk krb5kdc[912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449269, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:29.018677 942 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:54:29.019789 942 webserver.cc:460] Webserver started at http://127.3.221.126:44803/ using document root <none> and password file <none>
I20260811 11:54:29.020413 942 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:29.020491 942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:29.020709 942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:29.023068 942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "c3b62d0e0aac4ddeb09a6f7121be7aa3"
format_stamp: "Formatted at 2026-08-11 11:54:29 on dist-test-slave-bndk"
server_key: "6d5bc988d827793224965bd330231b9e"
server_key_iv: "a1be04a0181de19e247f3d1fbf0deced"
server_key_version: "encryptionkey@0"
I20260811 11:54:29.023716 942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "c3b62d0e0aac4ddeb09a6f7121be7aa3"
format_stamp: "Formatted at 2026-08-11 11:54:29 on dist-test-slave-bndk"
server_key: "6d5bc988d827793224965bd330231b9e"
server_key_iv: "a1be04a0181de19e247f3d1fbf0deced"
server_key_version: "encryptionkey@0"
I20260811 11:54:29.028258 942 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 11:54:29.031018 968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:29.032863 942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 11:54:29.033004 942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "c3b62d0e0aac4ddeb09a6f7121be7aa3"
format_stamp: "Formatted at 2026-08-11 11:54:29 on dist-test-slave-bndk"
server_key: "6d5bc988d827793224965bd330231b9e"
server_key_iv: "a1be04a0181de19e247f3d1fbf0deced"
server_key_version: "encryptionkey@0"
I20260811 11:54:29.033129 942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:29.050676 942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:29.054143 942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:29.054355 942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:29.064517 942 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:36279
I20260811 11:54:29.064975 1043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:36279 every 8 connection(s)
I20260811 11:54:29.066002 942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:54:29.069639 1045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:29.070925 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 942
I20260811 11:54:29.071067 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:54:29.071429 3957 external_mini_cluster.cc:1428] Setting key 4771e3a2f20d53180ebc71f91a0931b4
I20260811 11:54:29.079079 1045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3: Bootstrap starting.
Aug 11 11:54:29 dist-test-slave-bndk krb5kdc[912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:54:29.082000 1045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:29.082815 1045 log.cc:824] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:29.085193 1045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3: No bootstrap required, opened a new log
I20260811 11:54:29.086658 1050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:29.073073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38304 (local address 127.3.221.126:36279)
0811 11:54:29.073702 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:29.073721 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:29.073762 (+ 41us) server_negotiation.cc:407] Connection header received
0811 11:54:29.074502 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:29.074523 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:29.074934 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:29.075324 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:29.076266 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.077421 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.078357 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.078931 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.079106 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:29.081885 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:29.081891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:29.081907 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:29.081943 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:29.083804 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:29.084241 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:29.084247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:29.084253 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:29.084372 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:29.084638 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:29.084641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:29.084643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:29.085069 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:29.085288 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:29.085583 (+ 295us) server_negotiation.cc:299] Negotiation successful
0811 11:54:29.085900 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"spinlock_wait_cycles":38784,"thread_start_us":278,"threads_started":1}
I20260811 11:54:29.089618 1045 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b62d0e0aac4ddeb09a6f7121be7aa3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36279 } }
I20260811 11:54:29.089893 1045 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:29.089969 1045 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b62d0e0aac4ddeb09a6f7121be7aa3, State: Initialized, Role: FOLLOWER
I20260811 11:54:29.090415 1045 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b62d0e0aac4ddeb09a6f7121be7aa3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36279 } }
I20260811 11:54:29.090597 1045 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:54:29.090677 1045 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:54:29.090787 1045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:29.091965 1045 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b62d0e0aac4ddeb09a6f7121be7aa3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36279 } }
I20260811 11:54:29.092296 1045 leader_election.cc:302] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3b62d0e0aac4ddeb09a6f7121be7aa3; no voters:
I20260811 11:54:29.092586 1045 leader_election.cc:288] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:54:29.092712 1055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:29.092978 1055 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3 [term 1 LEADER]: Becoming Leader. State: Replica: c3b62d0e0aac4ddeb09a6f7121be7aa3, State: Running, Role: LEADER
I20260811 11:54:29.093343 1055 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b62d0e0aac4ddeb09a6f7121be7aa3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36279 } }
I20260811 11:54:29.093925 1045 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:54:29.095096 1059 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3b62d0e0aac4ddeb09a6f7121be7aa3. Latest consensus state: current_term: 1 leader_uuid: "c3b62d0e0aac4ddeb09a6f7121be7aa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b62d0e0aac4ddeb09a6f7121be7aa3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36279 } } }
I20260811 11:54:29.095252 1059 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:29.095113 1058 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3b62d0e0aac4ddeb09a6f7121be7aa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b62d0e0aac4ddeb09a6f7121be7aa3" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36279 } } }
I20260811 11:54:29.095415 1058 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:29.095729 1067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:54:29.099260 1067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:54:29.105506 1067 catalog_manager.cc:1347] Generated new cluster ID: aa1e5094cddf4b47b4fac2624f55ee77
I20260811 11:54:29.105602 1067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:54:29.117847 1067 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:54:29.118858 1067 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:54:29.123596 1067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3: Generated new TSK 0
I20260811 11:54:29.124313 1067 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
I20260811 11:54:29.162199 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36279
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:29.284560 1087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:29.284891 1087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:29.284955 1087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:29.289196 1087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:29.289318 1087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:29.289404 1087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:54:29.295385 1087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:29.296738 1087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:29.298240 1087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:29.309775 1103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:29.309763 1098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:29.309760 1101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:29.309861 1087 server_base.cc:1034] running on GCE node
I20260811 11:54:29.310691 1087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:29.311451 1087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:29.312776 1087 hybrid_clock.cc:648] HybridClock initialized: now 1786449269312745 us; error 58 us; skew 500 ppm
Aug 11 11:54:29 dist-test-slave-bndk krb5kdc[912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449269, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:29.316610 1087 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:54:29.317817 1087 webserver.cc:460] Webserver started at http://127.3.221.65:33165/ using document root <none> and password file <none>
I20260811 11:54:29.318465 1087 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:29.318547 1087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:29.318778 1087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:29.321130 1087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "f772310a876a42d3a8739170ab547d8a"
format_stamp: "Formatted at 2026-08-11 11:54:29 on dist-test-slave-bndk"
server_key: "69e36c3da6a9b75b31fa034cdfa33710"
server_key_iv: "e85edcd8b49f78be53805c4ae4c6e4a2"
server_key_version: "encryptionkey@0"
I20260811 11:54:29.321807 1087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "f772310a876a42d3a8739170ab547d8a"
format_stamp: "Formatted at 2026-08-11 11:54:29 on dist-test-slave-bndk"
server_key: "69e36c3da6a9b75b31fa034cdfa33710"
server_key_iv: "e85edcd8b49f78be53805c4ae4c6e4a2"
server_key_version: "encryptionkey@0"
I20260811 11:54:29.326510 1087 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:54:29.330425 1114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:29.331996 1087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20260811 11:54:29.332145 1087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "f772310a876a42d3a8739170ab547d8a"
format_stamp: "Formatted at 2026-08-11 11:54:29 on dist-test-slave-bndk"
server_key: "69e36c3da6a9b75b31fa034cdfa33710"
server_key_iv: "e85edcd8b49f78be53805c4ae4c6e4a2"
server_key_version: "encryptionkey@0"
I20260811 11:54:29.332300 1087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:29.347101 1087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:29.350533 1087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:29.350728 1087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:29.351402 1087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:54:29.352294 1087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:54:29.352352 1087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:29.352396 1087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:54:29.352423 1087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:29.361774 1087 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:41297
I20260811 11:54:29.361925 1278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:41297 every 8 connection(s)
I20260811 11:54:29.362954 1087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:54:29.371579 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 1087
I20260811 11:54:29.371843 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:54:29.372228 3957 external_mini_cluster.cc:1428] Setting key 43c946178c839d711bd02966f5891d3a
Aug 11 11:54:29 dist-test-slave-bndk krb5kdc[912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449269, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:54:29.378611 1050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:29.365060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:39189 (local address 127.3.221.126:36279)
0811 11:54:29.365285 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:29.365290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:29.366200 (+ 910us) server_negotiation.cc:407] Connection header received
0811 11:54:29.367572 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:29.367575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:29.367630 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:29.367711 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:29.369069 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.369707 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.370980 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.371240 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.371353 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:29.374335 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:29.374339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:29.374342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:29.374371 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:29.375709 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:29.376380 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:29.376384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:29.376386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:29.376512 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:29.377256 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:29.377259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:29.377261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:29.377528 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:29.377643 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:29.378298 (+ 655us) server_negotiation.cc:299] Negotiation successful
0811 11:54:29.378450 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260811 11:54:29.379638 1282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:29.365485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36279 (local address 127.3.221.65:39189)
0811 11:54:29.366010 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:29.366059 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:29.367315 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:29.367931 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:29.367940 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:29.368358 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:54:29.368884 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:29.368900 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.369849 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.369852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:29.370833 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:29.370846 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.371436 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.371441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:29.371662 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:29.372407 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:54:29.372425 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:54:29.374151 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:54:29.375843 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:54:29.375851 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:54:29.375870 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:54:29.376244 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:54:29.376752 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:54:29.376757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:54:29.376758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:54:29.377077 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:54:29.377707 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:54:29.377713 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:54:29.378014 (+ 301us) client_negotiation.cc:769] Sending connection context
0811 11:54:29.378220 (+ 206us) client_negotiation.cc:240] Negotiation successful
0811 11:54:29.378546 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":131,"threads_started":1}
I20260811 11:54:29.381003 1279 heartbeater.cc:347] Connected to a master server at 127.3.221.126:36279
I20260811 11:54:29.381363 1279 heartbeater.cc:464] Registering TS with master...
I20260811 11:54:29.382046 1279 heartbeater.cc:511] Master 127.3.221.126:36279 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.221.66@KRBTEST.COM; defaulting to no policy
I20260811 11:54:29.384048 991 ts_manager.cc:194] Registered new tserver with Master: f772310a876a42d3a8739170ab547d8a (127.3.221.65:41297)
I20260811 11:54:29.385344 991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:39189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
I20260811 11:54:29.404946 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36279
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:29.534459 1286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:29.534762 1286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:29.534807 1286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:29.539196 1286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:29.539281 1286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:29.539474 1286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:54:29.545531 1286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:29.546722 1286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:29.547789 1286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:29.558656 1291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:29.558656 1294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:29.558770 1297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:29.559265 1286 server_base.cc:1034] running on GCE node
I20260811 11:54:29.559751 1286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:29.560479 1286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:29.561873 1286 hybrid_clock.cc:648] HybridClock initialized: now 1786449269561844 us; error 63 us; skew 500 ppm
Aug 11 11:54:29 dist-test-slave-bndk krb5kdc[912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449269, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:29.565945 1286 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:54:29.567077 1286 webserver.cc:460] Webserver started at http://127.3.221.66:36301/ using document root <none> and password file <none>
I20260811 11:54:29.567716 1286 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:29.567792 1286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:29.568010 1286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:29.570489 1286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "83fb9825fdf1495c9fa388ecd464f50c"
format_stamp: "Formatted at 2026-08-11 11:54:29 on dist-test-slave-bndk"
server_key: "7a12e40c002d9f2f9c50d5c051a55706"
server_key_iv: "75e13bf1f3513c81c5eb648e450fe326"
server_key_version: "encryptionkey@0"
I20260811 11:54:29.571357 1286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "83fb9825fdf1495c9fa388ecd464f50c"
format_stamp: "Formatted at 2026-08-11 11:54:29 on dist-test-slave-bndk"
server_key: "7a12e40c002d9f2f9c50d5c051a55706"
server_key_iv: "75e13bf1f3513c81c5eb648e450fe326"
server_key_version: "encryptionkey@0"
I20260811 11:54:29.576483 1286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 11:54:29.579597 1317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:29.581164 1286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 11:54:29.581374 1286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "83fb9825fdf1495c9fa388ecd464f50c"
format_stamp: "Formatted at 2026-08-11 11:54:29 on dist-test-slave-bndk"
server_key: "7a12e40c002d9f2f9c50d5c051a55706"
server_key_iv: "75e13bf1f3513c81c5eb648e450fe326"
server_key_version: "encryptionkey@0"
I20260811 11:54:29.581501 1286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:29.597405 1286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:29.601862 1286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:29.602080 1286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:29.602746 1286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:54:29.603698 1286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:54:29.603752 1286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:29.603811 1286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:54:29.603837 1286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:29.614104 1286 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:36551
I20260811 11:54:29.614148 1489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:36551 every 8 connection(s)
I20260811 11:54:29.615113 1286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:54:29.624596 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 1286
I20260811 11:54:29.624732 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:54:29.625029 3957 external_mini_cluster.cc:1428] Setting key 5038ce262a07b505b67affea7b8f7d2c
Aug 11 11:54:29 dist-test-slave-bndk krb5kdc[912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449269, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:54:29.630895 1050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:29.617212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:37077 (local address 127.3.221.126:36279)
0811 11:54:29.617402 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:29.617407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:29.618374 (+ 967us) server_negotiation.cc:407] Connection header received
0811 11:54:29.619823 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:29.619827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:29.619884 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:29.619962 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:29.621561 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.622118 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.623469 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.623722 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.623850 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:29.626763 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:29.626768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:29.626771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:29.626799 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:29.628407 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:29.629027 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:29.629030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:29.629031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:29.629119 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:29.629518 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:29.629521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:29.629522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:29.629724 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:29.629816 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:29.630558 (+ 742us) server_negotiation.cc:299] Negotiation successful
0811 11:54:29.630709 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 11:54:29.631945 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:29.617671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36279 (local address 127.3.221.66:37077)
0811 11:54:29.618199 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:29.618245 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:29.619557 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:29.620186 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:29.620194 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:29.620753 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:54:29.621385 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:29.621405 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.622277 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.622280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:29.623304 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:29.623319 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.623929 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.623933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:29.624099 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:29.624750 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:54:29.624777 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:54:29.626589 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:54:29.628557 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:54:29.628564 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:54:29.628582 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:54:29.628888 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:54:29.629212 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:54:29.629242 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:54:29.629244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:54:29.629406 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:54:29.629832 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:54:29.629841 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:54:29.630227 (+ 386us) client_negotiation.cc:769] Sending connection context
0811 11:54:29.630450 (+ 223us) client_negotiation.cc:240] Negotiation successful
0811 11:54:29.630846 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":132,"threads_started":1}
I20260811 11:54:29.633239 1493 heartbeater.cc:347] Connected to a master server at 127.3.221.126:36279
I20260811 11:54:29.633540 1493 heartbeater.cc:464] Registering TS with master...
I20260811 11:54:29.634125 1493 heartbeater.cc:511] Master 127.3.221.126:36279 requested a full tablet report, sending...
I20260811 11:54:29.635227 991 ts_manager.cc:194] Registered new tserver with Master: 83fb9825fdf1495c9fa388ecd464f50c (127.3.221.66:36551)
I20260811 11:54:29.635808 991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:37077
WARNING: no policy specified for kudu/127.3.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
I20260811 11:54:29.661803 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36279
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:29.785369 1500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:29.785679 1500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:29.785742 1500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:29.789963 1500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:29.790041 1500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:29.790126 1500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:54:29.795922 1500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:29.797119 1500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:29.798208 1500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:54:29.808291 1500 server_base.cc:1034] running on GCE node
W20260811 11:54:29.808238 1507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:29.808442 1511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:29.808243 1508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:29.809011 1500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:29.809710 1500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:29.810873 1500 hybrid_clock.cc:648] HybridClock initialized: now 1786449269810855 us; error 28 us; skew 500 ppm
Aug 11 11:54:29 dist-test-slave-bndk krb5kdc[912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449269, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:29.814235 1500 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:54:29.815218 1500 webserver.cc:460] Webserver started at http://127.3.221.67:39997/ using document root <none> and password file <none>
I20260811 11:54:29.815814 1500 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:29.815886 1500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:29.816118 1500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:29.818408 1500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "f4eb263e3a1d420bb51d6045e48f3544"
format_stamp: "Formatted at 2026-08-11 11:54:29 on dist-test-slave-bndk"
server_key: "2b1c71e3525ed2ae366c5e8ba4bc4325"
server_key_iv: "21ebc8a90c41b3d81ffc9b987413a4b8"
server_key_version: "encryptionkey@0"
I20260811 11:54:29.819024 1500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "f4eb263e3a1d420bb51d6045e48f3544"
format_stamp: "Formatted at 2026-08-11 11:54:29 on dist-test-slave-bndk"
server_key: "2b1c71e3525ed2ae366c5e8ba4bc4325"
server_key_iv: "21ebc8a90c41b3d81ffc9b987413a4b8"
server_key_version: "encryptionkey@0"
I20260811 11:54:29.823088 1500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 11:54:29.825922 1523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:29.827405 1500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:54:29.827556 1500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "f4eb263e3a1d420bb51d6045e48f3544"
format_stamp: "Formatted at 2026-08-11 11:54:29 on dist-test-slave-bndk"
server_key: "2b1c71e3525ed2ae366c5e8ba4bc4325"
server_key_iv: "21ebc8a90c41b3d81ffc9b987413a4b8"
server_key_version: "encryptionkey@0"
I20260811 11:54:29.827680 1500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:29.852596 1500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:29.856217 1500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:29.856446 1500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:29.857106 1500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:54:29.858040 1500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:54:29.858093 1500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:29.858151 1500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:54:29.858180 1500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:29.867715 1500 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:40257
I20260811 11:54:29.867741 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:40257 every 8 connection(s)
I20260811 11:54:29.868690 1500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:54:29.870914 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 1500
I20260811 11:54:29.871188 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:54:29.871618 3957 external_mini_cluster.cc:1428] Setting key 01365bc97874f8841c4674a18e96690f
Aug 11 11:54:29 dist-test-slave-bndk krb5kdc[912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449269, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:54:29.883564 1050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:29.870824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:46017 (local address 127.3.221.126:36279)
0811 11:54:29.871189 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:29.871194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:29.871984 (+ 790us) server_negotiation.cc:407] Connection header received
0811 11:54:29.873594 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:29.873597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:29.873655 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:29.873734 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:29.875086 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.875654 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.877010 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.877312 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.877408 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:29.880023 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:29.880026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:29.880028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:29.880057 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:29.881364 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:29.881903 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:29.881906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:29.881907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:29.881990 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:29.882353 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:29.882356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:29.882357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:29.882566 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:29.882672 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:29.883297 (+ 625us) server_negotiation.cc:299] Negotiation successful
0811 11:54:29.883431 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 11:54:29.884472 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:29.871184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36279 (local address 127.3.221.67:46017)
0811 11:54:29.871789 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:29.871874 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:29.873392 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:29.873977 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:29.873985 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:29.874388 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:54:29.874919 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:29.874935 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.875810 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.875814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:29.876865 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:29.876877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.877469 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.877473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:29.877652 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:29.878227 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:54:29.878244 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:54:29.879816 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:54:29.881479 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:54:29.881485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:54:29.881499 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:54:29.881781 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:54:29.882068 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:54:29.882070 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:54:29.882072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:54:29.882217 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:54:29.882731 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:54:29.882742 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:54:29.883021 (+ 279us) client_negotiation.cc:769] Sending connection context
0811 11:54:29.883204 (+ 183us) client_negotiation.cc:240] Negotiation successful
0811 11:54:29.883494 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":144,"threads_started":1}
I20260811 11:54:29.885820 1712 heartbeater.cc:347] Connected to a master server at 127.3.221.126:36279
I20260811 11:54:29.886107 1712 heartbeater.cc:464] Registering TS with master...
I20260811 11:54:29.886662 1712 heartbeater.cc:511] Master 127.3.221.126:36279 requested a full tablet report, sending...
I20260811 11:54:29.887826 991 ts_manager.cc:194] Registered new tserver with Master: f4eb263e3a1d420bb51d6045e48f3544 (127.3.221.67:40257)
I20260811 11:54:29.888386 991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:46017
I20260811 11:54:29.897975 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:54:29.908192 1050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:29.900040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.3.221.126:36279)
0811 11:54:29.900225 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:29.900229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:29.900378 (+ 149us) server_negotiation.cc:407] Connection header received
0811 11:54:29.900536 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:29.900540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:29.900586 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:29.900653 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:29.901422 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.901968 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.902795 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.903021 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.903109 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:29.904780 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:29.904784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:29.904787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:29.904817 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:29.906236 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:29.906833 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:29.906836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:29.906837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:29.906921 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:29.907247 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:29.907251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:29.907253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:29.907499 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:29.907644 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:29.907932 (+ 288us) server_negotiation.cc:299] Negotiation successful
0811 11:54:29.908030 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260811 11:54:29.911496 991 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:54:29.914721 991 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:54:29.929651 1730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:29.924060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:41297 (local address 127.0.0.1:40264)
0811 11:54:29.924849 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:29.924860 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:29.924945 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:29.925509 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:29.925517 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:29.925544 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:29.925696 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:29.925703 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.926825 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.926828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:29.928072 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:29.928084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.929047 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.929051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:29.929163 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:29.929202 (+ 39us) client_negotiation.cc:769] Sending connection context
0811 11:54:29.929350 (+ 148us) client_negotiation.cc:240] Negotiation successful
0811 11:54:29.929495 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":108,"threads_started":1}
I20260811 11:54:29.929662 1733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:29.924517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40264 (local address 127.3.221.65:41297)
0811 11:54:29.924930 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:29.924936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:29.924961 (+ 25us) server_negotiation.cc:407] Connection header received
0811 11:54:29.925045 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:29.925050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:29.925326 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:29.925844 (+ 518us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:29.925968 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.926703 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.928221 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.928927 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.929026 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:29.929195 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:29.929388 (+ 193us) server_negotiation.cc:299] Negotiation successful
0811 11:54:29.929495 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":64,"threads_started":1}
I20260811 11:54:29.929960 1727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:29.923689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:36551 (local address 127.0.0.1:44150)
0811 11:54:29.924208 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:29.924263 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:29.924431 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:29.925643 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:29.925646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:29.925665 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:29.925765 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:29.925770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.927305 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.927308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:29.928443 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:29.928454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.929569 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.929573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:29.929663 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:29.929687 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 11:54:29.929737 (+ 50us) client_negotiation.cc:240] Negotiation successful
0811 11:54:29.929815 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":2,"spinlock_wait_cycles":3840,"thread_start_us":105,"threads_started":1}
I20260811 11:54:29.930331 1728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:29.923896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:40257 (local address 127.0.0.1:50844)
0811 11:54:29.924826 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:29.924840 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:29.924996 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:29.925680 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:29.925682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:29.925698 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:29.925796 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:29.925802 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.927338 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.927341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:29.928420 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:29.928431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.929893 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.929896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:29.929980 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:29.930003 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 11:54:29.930138 (+ 135us) client_negotiation.cc:240] Negotiation successful
0811 11:54:29.930209 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":48,"thread_start_us":123,"threads_started":1}
I20260811 11:54:29.930687 1731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:29.924396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44150 (local address 127.3.221.66:36551)
0811 11:54:29.925197 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:29.925209 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:29.925266 (+ 57us) server_negotiation.cc:407] Connection header received
0811 11:54:29.925334 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:29.925339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:29.925493 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:29.925892 (+ 399us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:29.926050 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.927142 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.928583 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.929425 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.930229 (+ 804us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:29.930392 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:29.930446 (+ 54us) server_negotiation.cc:299] Negotiation successful
0811 11:54:29.930541 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":109,"threads_started":1}
I20260811 11:54:29.930718 1732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:29.924408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.3.221.67:40257)
0811 11:54:29.924746 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:29.924758 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:29.924936 (+ 178us) server_negotiation.cc:407] Connection header received
0811 11:54:29.925138 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:29.925144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:29.925367 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:29.925495 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:29.925917 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.927142 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.928551 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:29.929425 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:29.929535 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:29.929677 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:29.930501 (+ 824us) server_negotiation.cc:299] Negotiation successful
0811 11:54:29.930591 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":132,"threads_started":1}
I20260811 11:54:29.933688 1391 tablet_service.cc:1467] Processing CreateTablet for tablet 92680edd917a4018a7551d8a4d1faadc (DEFAULT_TABLE table=test-table [id=08dd11b6595b4c2fb6b98a41cb5df222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:54:29.933826 1190 tablet_service.cc:1467] Processing CreateTablet for tablet 92680edd917a4018a7551d8a4d1faadc (DEFAULT_TABLE table=test-table [id=08dd11b6595b4c2fb6b98a41cb5df222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:54:29.934401 1589 tablet_service.cc:1467] Processing CreateTablet for tablet 92680edd917a4018a7551d8a4d1faadc (DEFAULT_TABLE table=test-table [id=08dd11b6595b4c2fb6b98a41cb5df222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:54:29.934981 1391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92680edd917a4018a7551d8a4d1faadc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:29.935251 1190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92680edd917a4018a7551d8a4d1faadc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:29.935364 1589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92680edd917a4018a7551d8a4d1faadc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:29.942211 1740 tablet_bootstrap.cc:492] T 92680edd917a4018a7551d8a4d1faadc P 83fb9825fdf1495c9fa388ecd464f50c: Bootstrap starting.
I20260811 11:54:29.943495 1741 tablet_bootstrap.cc:492] T 92680edd917a4018a7551d8a4d1faadc P f772310a876a42d3a8739170ab547d8a: Bootstrap starting.
I20260811 11:54:29.945104 1740 tablet_bootstrap.cc:654] T 92680edd917a4018a7551d8a4d1faadc P 83fb9825fdf1495c9fa388ecd464f50c: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:29.945415 1741 tablet_bootstrap.cc:654] T 92680edd917a4018a7551d8a4d1faadc P f772310a876a42d3a8739170ab547d8a: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:29.945619 1744 tablet_bootstrap.cc:492] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544: Bootstrap starting.
I20260811 11:54:29.945962 1740 log.cc:824] T 92680edd917a4018a7551d8a4d1faadc P 83fb9825fdf1495c9fa388ecd464f50c: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:29.946250 1741 log.cc:824] T 92680edd917a4018a7551d8a4d1faadc P f772310a876a42d3a8739170ab547d8a: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:29.948002 1744 tablet_bootstrap.cc:654] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:29.948308 1740 tablet_bootstrap.cc:492] T 92680edd917a4018a7551d8a4d1faadc P 83fb9825fdf1495c9fa388ecd464f50c: No bootstrap required, opened a new log
I20260811 11:54:29.948423 1741 tablet_bootstrap.cc:492] T 92680edd917a4018a7551d8a4d1faadc P f772310a876a42d3a8739170ab547d8a: No bootstrap required, opened a new log
I20260811 11:54:29.948516 1740 ts_tablet_manager.cc:1397] T 92680edd917a4018a7551d8a4d1faadc P 83fb9825fdf1495c9fa388ecd464f50c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 11:54:29.948623 1741 ts_tablet_manager.cc:1397] T 92680edd917a4018a7551d8a4d1faadc P f772310a876a42d3a8739170ab547d8a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260811 11:54:29.949400 1744 log.cc:824] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:29.951803 1744 tablet_bootstrap.cc:492] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544: No bootstrap required, opened a new log
I20260811 11:54:29.952026 1744 ts_tablet_manager.cc:1397] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260811 11:54:29.951848 1740 raft_consensus.cc:348] T 92680edd917a4018a7551d8a4d1faadc P 83fb9825fdf1495c9fa388ecd464f50c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fb9825fdf1495c9fa388ecd464f50c" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 36551 } } peers { permanent_uuid: "f4eb263e3a1d420bb51d6045e48f3544" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 40257 } } peers { permanent_uuid: "f772310a876a42d3a8739170ab547d8a" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41297 } }
I20260811 11:54:29.952124 1740 raft_consensus.cc:374] T 92680edd917a4018a7551d8a4d1faadc P 83fb9825fdf1495c9fa388ecd464f50c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:29.952178 1740 raft_consensus.cc:729] T 92680edd917a4018a7551d8a4d1faadc P 83fb9825fdf1495c9fa388ecd464f50c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83fb9825fdf1495c9fa388ecd464f50c, State: Initialized, Role: FOLLOWER
I20260811 11:54:29.952616 1740 consensus_queue.cc:260] T 92680edd917a4018a7551d8a4d1faadc P 83fb9825fdf1495c9fa388ecd464f50c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fb9825fdf1495c9fa388ecd464f50c" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 36551 } } peers { permanent_uuid: "f4eb263e3a1d420bb51d6045e48f3544" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 40257 } } peers { permanent_uuid: "f772310a876a42d3a8739170ab547d8a" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41297 } }
I20260811 11:54:29.953059 1741 raft_consensus.cc:348] T 92680edd917a4018a7551d8a4d1faadc P f772310a876a42d3a8739170ab547d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fb9825fdf1495c9fa388ecd464f50c" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 36551 } } peers { permanent_uuid: "f4eb263e3a1d420bb51d6045e48f3544" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 40257 } } peers { permanent_uuid: "f772310a876a42d3a8739170ab547d8a" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41297 } }
I20260811 11:54:29.953353 1741 raft_consensus.cc:374] T 92680edd917a4018a7551d8a4d1faadc P f772310a876a42d3a8739170ab547d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:29.953413 1741 raft_consensus.cc:729] T 92680edd917a4018a7551d8a4d1faadc P f772310a876a42d3a8739170ab547d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f772310a876a42d3a8739170ab547d8a, State: Initialized, Role: FOLLOWER
I20260811 11:54:29.953703 1740 ts_tablet_manager.cc:1428] T 92680edd917a4018a7551d8a4d1faadc P 83fb9825fdf1495c9fa388ecd464f50c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260811 11:54:29.953910 1741 consensus_queue.cc:260] T 92680edd917a4018a7551d8a4d1faadc P f772310a876a42d3a8739170ab547d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fb9825fdf1495c9fa388ecd464f50c" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 36551 } } peers { permanent_uuid: "f4eb263e3a1d420bb51d6045e48f3544" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 40257 } } peers { permanent_uuid: "f772310a876a42d3a8739170ab547d8a" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41297 } }
I20260811 11:54:29.954097 1493 heartbeater.cc:503] Master 127.3.221.126:36279 was elected leader, sending a full tablet report...
I20260811 11:54:29.954918 1741 ts_tablet_manager.cc:1428] T 92680edd917a4018a7551d8a4d1faadc P f772310a876a42d3a8739170ab547d8a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260811 11:54:29.955281 1279 heartbeater.cc:503] Master 127.3.221.126:36279 was elected leader, sending a full tablet report...
I20260811 11:54:29.955977 1744 raft_consensus.cc:348] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fb9825fdf1495c9fa388ecd464f50c" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 36551 } } peers { permanent_uuid: "f4eb263e3a1d420bb51d6045e48f3544" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 40257 } } peers { permanent_uuid: "f772310a876a42d3a8739170ab547d8a" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41297 } }
I20260811 11:54:29.956291 1744 raft_consensus.cc:374] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:29.956367 1744 raft_consensus.cc:729] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4eb263e3a1d420bb51d6045e48f3544, State: Initialized, Role: FOLLOWER
I20260811 11:54:29.956882 1744 consensus_queue.cc:260] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fb9825fdf1495c9fa388ecd464f50c" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 36551 } } peers { permanent_uuid: "f4eb263e3a1d420bb51d6045e48f3544" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 40257 } } peers { permanent_uuid: "f772310a876a42d3a8739170ab547d8a" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41297 } }
I20260811 11:54:29.957661 1712 heartbeater.cc:503] Master 127.3.221.126:36279 was elected leader, sending a full tablet report...
I20260811 11:54:29.957994 1744 ts_tablet_manager.cc:1428] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260811 11:54:30.099561 1752 raft_consensus.cc:482] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:54:30.099821 1752 raft_consensus.cc:504] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fb9825fdf1495c9fa388ecd464f50c" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 36551 } } peers { permanent_uuid: "f4eb263e3a1d420bb51d6045e48f3544" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 40257 } } peers { permanent_uuid: "f772310a876a42d3a8739170ab547d8a" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41297 } }
I20260811 11:54:30.101024 1752 leader_election.cc:288] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83fb9825fdf1495c9fa388ecd464f50c (127.3.221.66:36551), f772310a876a42d3a8739170ab547d8a (127.3.221.65:41297)
I20260811 11:54:30.104835 1731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:30.101410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:59055 (local address 127.3.221.66:36551)
0811 11:54:30.101548 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:30.101553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:30.101693 (+ 140us) server_negotiation.cc:407] Connection header received
0811 11:54:30.101860 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:30.101863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:30.101915 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:30.101989 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:30.102230 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:30.102730 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:30.103901 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:30.104291 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:30.104409 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:30.104466 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:30.104623 (+ 157us) server_negotiation.cc:299] Negotiation successful
0811 11:54:30.104698 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 11:54:30.104839 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:30.101332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:36551 (local address 127.3.221.67:59055)
0811 11:54:30.101556 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:30.101576 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:30.101699 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:30.102005 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:30.102008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:30.102021 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:30.102131 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:30.102136 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:30.102837 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:30.102840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:30.103759 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:30.103770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:30.104409 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:30.104412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:30.104513 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:30.104532 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 11:54:30.104599 (+ 67us) client_negotiation.cc:240] Negotiation successful
0811 11:54:30.104690 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":76}
I20260811 11:54:30.104874 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:30.101382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:41297 (local address 127.3.221.67:40763)
0811 11:54:30.101737 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:30.101768 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:30.101867 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:30.102117 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:30.102120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:30.102138 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:30.102250 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:30.102255 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:30.102955 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:30.102958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:30.103808 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:30.103817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:30.104455 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:30.104459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:30.104558 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:30.104577 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 11:54:30.104641 (+ 64us) client_negotiation.cc:240] Negotiation successful
0811 11:54:30.104712 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":40,"thread_start_us":157,"threads_started":1}
I20260811 11:54:30.104861 1733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:30.101773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:40763 (local address 127.3.221.65:41297)
0811 11:54:30.101913 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:30.101916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:30.101929 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:54:30.101976 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:30.101978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:30.102015 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:30.102082 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:30.102363 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:30.102849 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:30.103928 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:30.104341 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:30.104450 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:30.104500 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:30.104666 (+ 166us) server_negotiation.cc:299] Negotiation successful
0811 11:54:30.104737 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 11:54:30.105618 1215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92680edd917a4018a7551d8a4d1faadc" candidate_uuid: "f4eb263e3a1d420bb51d6045e48f3544" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f772310a876a42d3a8739170ab547d8a" is_pre_election: true
I20260811 11:54:30.105615 1423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92680edd917a4018a7551d8a4d1faadc" candidate_uuid: "f4eb263e3a1d420bb51d6045e48f3544" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83fb9825fdf1495c9fa388ecd464f50c" is_pre_election: true
I20260811 11:54:30.105911 1215 raft_consensus.cc:2454] T 92680edd917a4018a7551d8a4d1faadc P f772310a876a42d3a8739170ab547d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4eb263e3a1d420bb51d6045e48f3544 in term 0.
I20260811 11:54:30.105921 1423 raft_consensus.cc:2454] T 92680edd917a4018a7551d8a4d1faadc P 83fb9825fdf1495c9fa388ecd464f50c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4eb263e3a1d420bb51d6045e48f3544 in term 0.
I20260811 11:54:30.106400 1525 leader_election.cc:302] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83fb9825fdf1495c9fa388ecd464f50c, f4eb263e3a1d420bb51d6045e48f3544; no voters:
I20260811 11:54:30.106709 1752 raft_consensus.cc:2781] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:54:30.106791 1752 raft_consensus.cc:482] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:54:30.106823 1752 raft_consensus.cc:3037] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:30.107904 1752 raft_consensus.cc:504] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fb9825fdf1495c9fa388ecd464f50c" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 36551 } } peers { permanent_uuid: "f4eb263e3a1d420bb51d6045e48f3544" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 40257 } } peers { permanent_uuid: "f772310a876a42d3a8739170ab547d8a" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41297 } }
I20260811 11:54:30.108281 1752 leader_election.cc:288] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544 [CANDIDATE]: Term 1 election: Requested vote from peers 83fb9825fdf1495c9fa388ecd464f50c (127.3.221.66:36551), f772310a876a42d3a8739170ab547d8a (127.3.221.65:41297)
I20260811 11:54:30.108705 1215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92680edd917a4018a7551d8a4d1faadc" candidate_uuid: "f4eb263e3a1d420bb51d6045e48f3544" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f772310a876a42d3a8739170ab547d8a"
I20260811 11:54:30.108724 1423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92680edd917a4018a7551d8a4d1faadc" candidate_uuid: "f4eb263e3a1d420bb51d6045e48f3544" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83fb9825fdf1495c9fa388ecd464f50c"
I20260811 11:54:30.108852 1423 raft_consensus.cc:3037] T 92680edd917a4018a7551d8a4d1faadc P 83fb9825fdf1495c9fa388ecd464f50c [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:30.108853 1215 raft_consensus.cc:3037] T 92680edd917a4018a7551d8a4d1faadc P f772310a876a42d3a8739170ab547d8a [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:30.110203 1423 raft_consensus.cc:2454] T 92680edd917a4018a7551d8a4d1faadc P 83fb9825fdf1495c9fa388ecd464f50c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4eb263e3a1d420bb51d6045e48f3544 in term 1.
I20260811 11:54:30.110203 1215 raft_consensus.cc:2454] T 92680edd917a4018a7551d8a4d1faadc P f772310a876a42d3a8739170ab547d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4eb263e3a1d420bb51d6045e48f3544 in term 1.
I20260811 11:54:30.110723 1525 leader_election.cc:302] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83fb9825fdf1495c9fa388ecd464f50c, f4eb263e3a1d420bb51d6045e48f3544; no voters:
I20260811 11:54:30.110911 1752 raft_consensus.cc:2781] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:30.111164 1752 raft_consensus.cc:686] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544 [term 1 LEADER]: Becoming Leader. State: Replica: f4eb263e3a1d420bb51d6045e48f3544, State: Running, Role: LEADER
I20260811 11:54:30.111578 1752 consensus_queue.cc:237] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fb9825fdf1495c9fa388ecd464f50c" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 36551 } } peers { permanent_uuid: "f4eb263e3a1d420bb51d6045e48f3544" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 40257 } } peers { permanent_uuid: "f772310a876a42d3a8739170ab547d8a" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41297 } }
W20260811 11:54:30.114832 1280 tablet.cc:2369] T 92680edd917a4018a7551d8a4d1faadc P f772310a876a42d3a8739170ab547d8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:54:30.115693 989 catalog_manager.cc:5495] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4eb263e3a1d420bb51d6045e48f3544 (127.3.221.67). New cstate: current_term: 1 leader_uuid: "f4eb263e3a1d420bb51d6045e48f3544" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fb9825fdf1495c9fa388ecd464f50c" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4eb263e3a1d420bb51d6045e48f3544" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 40257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f772310a876a42d3a8739170ab547d8a" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41297 } health_report { overall_health: UNKNOWN } } }
W20260811 11:54:30.116353 1495 tablet.cc:2369] T 92680edd917a4018a7551d8a4d1faadc P 83fb9825fdf1495c9fa388ecd464f50c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:54:30.119692 1713 tablet.cc:2369] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:54:30.173512 1732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:30.169682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.3.221.67:40257)
0811 11:54:30.169822 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:30.169827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:30.169842 (+ 15us) server_negotiation.cc:407] Connection header received
0811 11:54:30.169924 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:30.169927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:30.169982 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:30.170078 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:54:30.170327 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:30.170832 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:30.171691 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:30.171904 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:30.171994 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:30.172323 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:54:30.172946 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:54:30.173118 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:30.173284 (+ 166us) server_negotiation.cc:299] Negotiation successful
0811 11:54:30.173330 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 11:54:30.180014 1215 raft_consensus.cc:1261] T 92680edd917a4018a7551d8a4d1faadc P f772310a876a42d3a8739170ab547d8a [term 1 FOLLOWER]: Refusing update from remote peer f4eb263e3a1d420bb51d6045e48f3544: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:54:30.180014 1423 raft_consensus.cc:1261] T 92680edd917a4018a7551d8a4d1faadc P 83fb9825fdf1495c9fa388ecd464f50c [term 1 FOLLOWER]: Refusing update from remote peer f4eb263e3a1d420bb51d6045e48f3544: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:54:30.180773 1752 consensus_queue.cc:1035] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83fb9825fdf1495c9fa388ecd464f50c" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:54:30.180999 1755 consensus_queue.cc:1035] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f772310a876a42d3a8739170ab547d8a" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 41297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:54:30.190661 1763 mvcc.cc:205] Tried to move back new op lower bound from 7317296210649460736 to 7317296210380214272. Current Snapshot: MvccSnapshot[applied={T|T < 7317296210649460736}]
I20260811 11:54:30.191140 1761 mvcc.cc:205] Tried to move back new op lower bound from 7317296210649460736 to 7317296210380214272. Current Snapshot: MvccSnapshot[applied={T|T < 7317296210649460736}]
I20260811 11:54:30.200956 989 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:54:30.201143 989 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:54:30.203859 989 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3b62d0e0aac4ddeb09a6f7121be7aa3: Sending DeleteTablet for 3 replicas of tablet 92680edd917a4018a7551d8a4d1faadc
I20260811 11:54:30.204759 1391 tablet_service.cc:1514] Processing DeleteTablet for tablet 92680edd917a4018a7551d8a4d1faadc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:54:30 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:44150
I20260811 11:54:30.204751 1190 tablet_service.cc:1514] Processing DeleteTablet for tablet 92680edd917a4018a7551d8a4d1faadc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:54:30 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:40264
I20260811 11:54:30.204802 1589 tablet_service.cc:1514] Processing DeleteTablet for tablet 92680edd917a4018a7551d8a4d1faadc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:54:30 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:50844
I20260811 11:54:30.205544 1774 tablet_replica.cc:330] T 92680edd917a4018a7551d8a4d1faadc P 83fb9825fdf1495c9fa388ecd464f50c: stopping tablet replica
I20260811 11:54:30.205905 1774 raft_consensus.cc:2229] T 92680edd917a4018a7551d8a4d1faadc P 83fb9825fdf1495c9fa388ecd464f50c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:54:30.206282 1774 raft_consensus.cc:2258] T 92680edd917a4018a7551d8a4d1faadc P 83fb9825fdf1495c9fa388ecd464f50c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:54:30.206346 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 1087
I20260811 11:54:30.217975 1776 tablet_replica.cc:330] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544: stopping tablet replica
I20260811 11:54:30.218499 1776 raft_consensus.cc:2229] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:54:30.219122 1776 raft_consensus.cc:2258] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:54:30.221439 1774 ts_tablet_manager.cc:1905] T 92680edd917a4018a7551d8a4d1faadc P 83fb9825fdf1495c9fa388ecd464f50c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:54:30.222827 1776 ts_tablet_manager.cc:1905] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:54:30.224465 1774 ts_tablet_manager.cc:1918] T 92680edd917a4018a7551d8a4d1faadc P 83fb9825fdf1495c9fa388ecd464f50c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:54:30.224578 1774 log.cc:1196] T 92680edd917a4018a7551d8a4d1faadc P 83fb9825fdf1495c9fa388ecd464f50c: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/92680edd917a4018a7551d8a4d1faadc
I20260811 11:54:30.224884 1774 ts_tablet_manager.cc:1939] T 92680edd917a4018a7551d8a4d1faadc P 83fb9825fdf1495c9fa388ecd464f50c: Deleting consensus metadata
I20260811 11:54:30.226325 971 catalog_manager.cc:4841] TS 83fb9825fdf1495c9fa388ecd464f50c (127.3.221.66:36551): tablet 92680edd917a4018a7551d8a4d1faadc (table test-table [id=08dd11b6595b4c2fb6b98a41cb5df222]) successfully deleted
I20260811 11:54:30.227169 1776 ts_tablet_manager.cc:1918] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:54:30.227288 1776 log.cc:1196] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/92680edd917a4018a7551d8a4d1faadc
I20260811 11:54:30.227636 1776 ts_tablet_manager.cc:1939] T 92680edd917a4018a7551d8a4d1faadc P f4eb263e3a1d420bb51d6045e48f3544: Deleting consensus metadata
W20260811 11:54:30.228556 969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.221.65:41297 (error 108)
I20260811 11:54:30.229061 973 catalog_manager.cc:4841] TS f4eb263e3a1d420bb51d6045e48f3544 (127.3.221.67:40257): tablet 92680edd917a4018a7551d8a4d1faadc (table test-table [id=08dd11b6595b4c2fb6b98a41cb5df222]) successfully deleted
I20260811 11:54:30.229597 1728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:30.229315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:41297 (local address 127.0.0.1:40268)
0811 11:54:30.229445 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:30.229512 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.65:41297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260811 11:54:30.229786 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 1286
W20260811 11:54:30.229779 969 catalog_manager.cc:4584] TS f772310a876a42d3a8739170ab547d8a (127.3.221.65:41297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 92680edd917a4018a7551d8a4d1faadc: Network error: Client connection negotiation failed: client connection to 127.3.221.65:41297: connect: Connection refused (error 111)
I20260811 11:54:30.277065 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 1500
I20260811 11:54:30.293448 1728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:30.284814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:41297 (local address 127.0.0.1:40280)
0811 11:54:30.293204 (+ 8390us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:30.293331 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.65:41297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8240}
I20260811 11:54:30.294894 1728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:30.294412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:41297 (local address 127.0.0.1:40296)
0811 11:54:30.294686 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:30.294817 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.65:41297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260811 11:54:30.300823 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 942
2026-08-11T11:54:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2788 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-11T11:54:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:54:30Z Disabled control of system clock
I20260811 11:54:30.345476 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:40419
--webserver_interface=127.3.221.126
--webserver_port=0
--builtin_ntp_servers=127.3.221.84:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:40419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 11:54:30.471629 1783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:30.471959 1783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:30.471997 1783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:30.476265 1783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:54:30.476344 1783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:30.476359 1783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:30.476375 1783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:54:30.476389 1783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:54:30.481918 1783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:40419
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:40419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:30.483229 1783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:30.484712 1783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:30.494967 1793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:30.495124 1783 server_base.cc:1034] running on GCE node
W20260811 11:54:30.495023 1795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:30.494967 1799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:30.495939 1783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:30.497159 1783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:30.498407 1783 hybrid_clock.cc:648] HybridClock initialized: now 1786449270498359 us; error 59 us; skew 500 ppm
I20260811 11:54:30.500590 1783 webserver.cc:460] Webserver started at http://127.3.221.126:34415/ using document root <none> and password file <none>
I20260811 11:54:30.501322 1783 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:30.501411 1783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:30.501660 1783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:30.504011 1783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "9c5017af30b8414889ab8e647f6979b4"
format_stamp: "Formatted at 2026-08-11 11:54:30 on dist-test-slave-bndk"
server_key: "7b048c5cef56ac05f3b3fdc591d5fb47"
server_key_iv: "47cf524d9d2bf3f1140ffb71738b27b7"
server_key_version: "encryptionkey@0"
I20260811 11:54:30.504634 1783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "9c5017af30b8414889ab8e647f6979b4"
format_stamp: "Formatted at 2026-08-11 11:54:30 on dist-test-slave-bndk"
server_key: "7b048c5cef56ac05f3b3fdc591d5fb47"
server_key_iv: "47cf524d9d2bf3f1140ffb71738b27b7"
server_key_version: "encryptionkey@0"
I20260811 11:54:30.508977 1783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260811 11:54:30.511915 1807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:30.513514 1783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:54:30.513659 1783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "9c5017af30b8414889ab8e647f6979b4"
format_stamp: "Formatted at 2026-08-11 11:54:30 on dist-test-slave-bndk"
server_key: "7b048c5cef56ac05f3b3fdc591d5fb47"
server_key_iv: "47cf524d9d2bf3f1140ffb71738b27b7"
server_key_version: "encryptionkey@0"
I20260811 11:54:30.513780 1783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:30.525988 1783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:30.526796 1783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:30.526993 1783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:30.535199 1783 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:40419
I20260811 11:54:30.535411 1892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:40419 every 8 connection(s)
I20260811 11:54:30.536343 1783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:54:30.540640 1893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:30.543792 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 1783
I20260811 11:54:30.543915 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:54:30.544188 3957 external_mini_cluster.cc:1428] Setting key 512ea676c57c862fd999d7efbbffd16d
I20260811 11:54:30.547402 1893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4: Bootstrap starting.
I20260811 11:54:30.550197 1893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:30.551023 1893 log.cc:824] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:30.553496 1893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4: No bootstrap required, opened a new log
I20260811 11:54:30.556464 1897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:30.546379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54144 (local address 127.3.221.126:40419)
0811 11:54:30.547205 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:30.547227 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:30.547268 (+ 41us) server_negotiation.cc:407] Connection header received
0811 11:54:30.548398 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:30.548440 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:30.549129 (+ 689us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:30.549689 (+ 560us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:30.550174 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:30.551511 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:30.552779 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:30.553364 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:30.553500 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:30.554416 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:30.554421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:30.554437 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:30.554854 (+ 417us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:30.554916 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:30.554938 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:30.555018 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:30.555181 (+ 163us) server_negotiation.cc:299] Negotiation successful
0811 11:54:30.555481 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":191,"threads_started":1}
I20260811 11:54:30.557039 1893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c5017af30b8414889ab8e647f6979b4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } }
I20260811 11:54:30.557314 1893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:30.557374 1893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c5017af30b8414889ab8e647f6979b4, State: Initialized, Role: FOLLOWER
I20260811 11:54:30.557837 1893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c5017af30b8414889ab8e647f6979b4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } }
I20260811 11:54:30.557969 1893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:54:30.558030 1893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:54:30.558130 1893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:30.559115 1893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c5017af30b8414889ab8e647f6979b4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } }
I20260811 11:54:30.559464 1893 leader_election.cc:302] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c5017af30b8414889ab8e647f6979b4; no voters:
I20260811 11:54:30.559788 1893 leader_election.cc:288] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:54:30.559912 1899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:30.560153 1899 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4 [term 1 LEADER]: Becoming Leader. State: Replica: 9c5017af30b8414889ab8e647f6979b4, State: Running, Role: LEADER
I20260811 11:54:30.560475 1899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c5017af30b8414889ab8e647f6979b4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } }
I20260811 11:54:30.561034 1893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:54:30.562242 1900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c5017af30b8414889ab8e647f6979b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c5017af30b8414889ab8e647f6979b4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } } }
I20260811 11:54:30.562395 1900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:30.562705 1901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c5017af30b8414889ab8e647f6979b4. Latest consensus state: current_term: 1 leader_uuid: "9c5017af30b8414889ab8e647f6979b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c5017af30b8414889ab8e647f6979b4" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 40419 } } }
I20260811 11:54:30.562810 1901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:30.563666 1912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:54:30.566817 1912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:54:30.572790 1912 catalog_manager.cc:1347] Generated new cluster ID: c2b91f39447b433b86d91945eb1889d0
I20260811 11:54:30.572885 1912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:54:30.588104 1912 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:54:30.589581 1912 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:54:30.605880 1912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4: Generated new TSK 0
I20260811 11:54:30.607017 1912 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:54:30.614301 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:40419
--builtin_ntp_servers=127.3.221.84:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 11:54:30.743600 1931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:30.743899 1931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:30.743944 1931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:30.748477 1931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:30.748574 1931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:30.748634 1931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:54:30.753912 1931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:40419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:30.755313 1931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:30.756927 1931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:30.767670 1937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:30.767663 1938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:30.767668 1941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:30.768280 1931 server_base.cc:1034] running on GCE node
I20260811 11:54:30.768754 1931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:30.769503 1931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:30.770691 1931 hybrid_clock.cc:648] HybridClock initialized: now 1786449270770664 us; error 43 us; skew 500 ppm
I20260811 11:54:30.773300 1931 webserver.cc:460] Webserver started at http://127.3.221.65:46315/ using document root <none> and password file <none>
I20260811 11:54:30.773934 1931 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:30.774003 1931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:30.774212 1931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:30.776459 1931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "2713053d46714c879a8057674ddff964"
format_stamp: "Formatted at 2026-08-11 11:54:30 on dist-test-slave-bndk"
server_key: "dc389cd82aee4357c220229f080e4d94"
server_key_iv: "733edcad955e0bf72c4fef0d1fd2b39b"
server_key_version: "encryptionkey@0"
I20260811 11:54:30.777053 1931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "2713053d46714c879a8057674ddff964"
format_stamp: "Formatted at 2026-08-11 11:54:30 on dist-test-slave-bndk"
server_key: "dc389cd82aee4357c220229f080e4d94"
server_key_iv: "733edcad955e0bf72c4fef0d1fd2b39b"
server_key_version: "encryptionkey@0"
I20260811 11:54:30.781034 1931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260811 11:54:30.783776 1953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:30.785151 1931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 11:54:30.785346 1931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "2713053d46714c879a8057674ddff964"
format_stamp: "Formatted at 2026-08-11 11:54:30 on dist-test-slave-bndk"
server_key: "dc389cd82aee4357c220229f080e4d94"
server_key_iv: "733edcad955e0bf72c4fef0d1fd2b39b"
server_key_version: "encryptionkey@0"
I20260811 11:54:30.785467 1931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:30.798920 1931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:30.799712 1931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:30.799907 1931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:30.800576 1931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:54:30.801668 1931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:54:30.801719 1931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:30.801779 1931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:54:30.801803 1931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:30.810755 1931 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:33059
I20260811 11:54:30.810842 2156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:33059 every 8 connection(s)
I20260811 11:54:30.811734 1931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:54:30.813361 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 1931
I20260811 11:54:30.813476 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:54:30.813791 3957 external_mini_cluster.cc:1428] Setting key f612b6f200c4697de80a08b5222467be
I20260811 11:54:30.816098 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:40419
--builtin_ntp_servers=127.3.221.84:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 11:54:30.823292 1897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:30.813700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:45527 (local address 127.3.221.126:40419)
0811 11:54:30.813847 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:30.813852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:30.815192 (+ 1340us) server_negotiation.cc:407] Connection header received
0811 11:54:30.817072 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:30.817075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:30.817125 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:30.817200 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:30.818535 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:30.819355 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:30.821020 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:30.821294 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:30.821401 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:30.822803 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:30.822807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:30.822809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:30.822823 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:30.822843 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:30.822851 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:30.822920 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:30.823101 (+ 181us) server_negotiation.cc:299] Negotiation successful
0811 11:54:30.823161 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 11:54:30.824606 2159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:30.814138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.65:45527)
0811 11:54:30.815025 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:30.815085 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:30.816803 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:30.817480 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:30.817489 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:30.817610 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:54:30.818323 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:30.818340 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:30.819520 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:30.819523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:30.820852 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:30.820866 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:30.821517 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:30.821523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:30.821740 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:30.822449 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:54:30.822451 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:54:30.822514 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:54:30.822615 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:54:30.822943 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:54:30.822953 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:54:30.822960 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 11:54:30.823108 (+ 148us) client_negotiation.cc:240] Negotiation successful
0811 11:54:30.823468 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":158,"threads_started":1}
I20260811 11:54:30.826021 2157 heartbeater.cc:347] Connected to a master server at 127.3.221.126:40419
I20260811 11:54:30.826278 2157 heartbeater.cc:464] Registering TS with master...
I20260811 11:54:30.826850 2157 heartbeater.cc:511] Master 127.3.221.126:40419 requested a full tablet report, sending...
I20260811 11:54:30.828447 1831 ts_manager.cc:194] Registered new tserver with Master: 2713053d46714c879a8057674ddff964 (127.3.221.65:33059)
I20260811 11:54:30.829689 1831 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.221.65:45527
W20260811 11:54:30.937994 2160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:30.938313 2160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:30.938377 2160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:30.942566 2160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:30.942643 2160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:30.942724 2160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:54:30.947985 2160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:40419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:30.949277 2160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:30.950762 2160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:30.961154 2167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:30.961336 2160 server_base.cc:1034] running on GCE node
W20260811 11:54:30.961256 2170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:30.961237 2166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:30.961954 2160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:30.962661 2160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:30.963836 2160 hybrid_clock.cc:648] HybridClock initialized: now 1786449270963822 us; error 28 us; skew 500 ppm
I20260811 11:54:30.966069 2160 webserver.cc:460] Webserver started at http://127.3.221.66:34329/ using document root <none> and password file <none>
I20260811 11:54:30.966722 2160 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:30.966796 2160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:30.967007 2160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:30.969285 2160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "ebcb585c28d54c239bfc7b1fe71302ba"
format_stamp: "Formatted at 2026-08-11 11:54:30 on dist-test-slave-bndk"
server_key: "108772426995adf7e0d4bdfa129ccc99"
server_key_iv: "1c900e8ea39bd14353f2b95da9f12467"
server_key_version: "encryptionkey@0"
I20260811 11:54:30.969908 2160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "ebcb585c28d54c239bfc7b1fe71302ba"
format_stamp: "Formatted at 2026-08-11 11:54:30 on dist-test-slave-bndk"
server_key: "108772426995adf7e0d4bdfa129ccc99"
server_key_iv: "1c900e8ea39bd14353f2b95da9f12467"
server_key_version: "encryptionkey@0"
I20260811 11:54:30.973985 2160 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 11:54:30.976825 2180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:30.978508 2160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:54:30.978644 2160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "ebcb585c28d54c239bfc7b1fe71302ba"
format_stamp: "Formatted at 2026-08-11 11:54:30 on dist-test-slave-bndk"
server_key: "108772426995adf7e0d4bdfa129ccc99"
server_key_iv: "1c900e8ea39bd14353f2b95da9f12467"
server_key_version: "encryptionkey@0"
I20260811 11:54:30.978768 2160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:30.990641 2160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:30.991420 2160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:30.991611 2160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:30.992242 2160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:54:30.993294 2160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:54:30.993351 2160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:30.993409 2160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:54:30.993436 2160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:31.002718 2160 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:40221
I20260811 11:54:31.002810 2369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:40221 every 8 connection(s)
I20260811 11:54:31.003682 2160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:54:31.005533 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 2160
I20260811 11:54:31.005654 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:54:31.006006 3957 external_mini_cluster.cc:1428] Setting key 3aad586843bf87ddcafe97d038b6e6b3
I20260811 11:54:31.007977 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:40419
--builtin_ntp_servers=127.3.221.84:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 11:54:31.014024 1897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:31.005876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:33601 (local address 127.3.221.126:40419)
0811 11:54:31.006031 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:31.006036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:31.007263 (+ 1227us) server_negotiation.cc:407] Connection header received
0811 11:54:31.008785 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:31.008788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:31.008844 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:31.008929 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:31.010118 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.010680 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.012075 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.012302 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.012398 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:31.013545 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:31.013549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:31.013551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:31.013566 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:31.013585 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:31.013593 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:31.013653 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:31.013800 (+ 147us) server_negotiation.cc:299] Negotiation successful
0811 11:54:31.013872 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 11:54:31.014935 2372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:31.006341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.66:33601)
0811 11:54:31.007099 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:31.007157 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:31.008551 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:31.009086 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:31.009094 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:31.009183 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:54:31.009904 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:31.009933 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.010845 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.010849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:31.011942 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:31.011954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.012458 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.012463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:31.012628 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:31.013257 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:54:31.013260 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:54:31.013311 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:54:31.013388 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:54:31.013663 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:54:31.013673 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:54:31.013680 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 11:54:31.013792 (+ 112us) client_negotiation.cc:240] Negotiation successful
0811 11:54:31.014098 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":125,"threads_started":1}
I20260811 11:54:31.016296 2370 heartbeater.cc:347] Connected to a master server at 127.3.221.126:40419
I20260811 11:54:31.016556 2370 heartbeater.cc:464] Registering TS with master...
I20260811 11:54:31.017165 2370 heartbeater.cc:511] Master 127.3.221.126:40419 requested a full tablet report, sending...
I20260811 11:54:31.018414 1831 ts_manager.cc:194] Registered new tserver with Master: ebcb585c28d54c239bfc7b1fe71302ba (127.3.221.66:40221)
I20260811 11:54:31.019217 1831 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.221.66:33601
W20260811 11:54:31.129722 2373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:31.130033 2373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:31.130074 2373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:31.134389 2373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:31.134471 2373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:31.134536 2373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:54:31.139698 2373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:40419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:31.140856 2373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:31.142298 2373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:31.153349 2381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:31.153340 2383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:31.153332 2380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:31.153419 2373 server_base.cc:1034] running on GCE node
I20260811 11:54:31.154057 2373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:31.154743 2373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:31.155951 2373 hybrid_clock.cc:648] HybridClock initialized: now 1786449271155930 us; error 35 us; skew 500 ppm
I20260811 11:54:31.158263 2373 webserver.cc:460] Webserver started at http://127.3.221.67:41889/ using document root <none> and password file <none>
I20260811 11:54:31.158895 2373 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:31.158964 2373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:31.159169 2373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:31.161597 2373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "fedcee92d3bc48b1b27d6dc63820aa3d"
format_stamp: "Formatted at 2026-08-11 11:54:31 on dist-test-slave-bndk"
server_key: "1ec89e75c0c15b86aa5f7c7062343020"
server_key_iv: "7f6a16829063d8681167b171935c80c9"
server_key_version: "encryptionkey@0"
I20260811 11:54:31.162225 2373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "fedcee92d3bc48b1b27d6dc63820aa3d"
format_stamp: "Formatted at 2026-08-11 11:54:31 on dist-test-slave-bndk"
server_key: "1ec89e75c0c15b86aa5f7c7062343020"
server_key_iv: "7f6a16829063d8681167b171935c80c9"
server_key_version: "encryptionkey@0"
I20260811 11:54:31.166527 2373 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:54:31.169440 2392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:31.170895 2373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:54:31.171044 2373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "fedcee92d3bc48b1b27d6dc63820aa3d"
format_stamp: "Formatted at 2026-08-11 11:54:31 on dist-test-slave-bndk"
server_key: "1ec89e75c0c15b86aa5f7c7062343020"
server_key_iv: "7f6a16829063d8681167b171935c80c9"
server_key_version: "encryptionkey@0"
I20260811 11:54:31.171169 2373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:31.181375 2373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:31.182185 2373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:31.182374 2373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:31.183010 2373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:54:31.184247 2373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:54:31.184305 2373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:31.184371 2373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:54:31.184425 2373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:31.194180 2373 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:44323
I20260811 11:54:31.194199 2574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:44323 every 8 connection(s)
I20260811 11:54:31.195218 2373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:54:31.196441 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 2373
I20260811 11:54:31.196533 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:54:31.196794 3957 external_mini_cluster.cc:1428] Setting key 34e2b45feaeb71ac8075565a481e1a0a
I20260811 11:54:31.204793 1897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:31.197181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:40521 (local address 127.3.221.126:40419)
0811 11:54:31.197438 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:31.197443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:31.198639 (+ 1196us) server_negotiation.cc:407] Connection header received
0811 11:54:31.200099 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:31.200102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:31.200161 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:31.200233 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:31.201156 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.201758 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.202904 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.203134 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.203223 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:31.204315 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:31.204318 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:31.204322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:31.204337 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:31.204355 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:31.204363 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:31.204433 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:31.204569 (+ 136us) server_negotiation.cc:299] Negotiation successful
0811 11:54:31.204638 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260811 11:54:31.205729 2579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:31.197760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:40419 (local address 127.3.221.67:40521)
0811 11:54:31.198500 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:31.198544 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:31.199879 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:31.200413 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:31.200420 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:31.200505 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:54:31.201017 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:31.201030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.201888 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.201891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:31.202779 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:31.202790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.203281 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.203285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:31.203450 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:31.204057 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:54:31.204058 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:54:31.204100 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:54:31.204169 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:54:31.204435 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:54:31.204444 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:54:31.204451 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 11:54:31.204563 (+ 112us) client_negotiation.cc:240] Negotiation successful
0811 11:54:31.204869 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":128,"threads_started":1}
I20260811 11:54:31.206995 2575 heartbeater.cc:347] Connected to a master server at 127.3.221.126:40419
I20260811 11:54:31.207266 2575 heartbeater.cc:464] Registering TS with master...
I20260811 11:54:31.207835 2575 heartbeater.cc:511] Master 127.3.221.126:40419 requested a full tablet report, sending...
I20260811 11:54:31.208995 1831 ts_manager.cc:194] Registered new tserver with Master: fedcee92d3bc48b1b27d6dc63820aa3d (127.3.221.67:44323)
I20260811 11:54:31.209699 1831 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.221.67:40521
I20260811 11:54:31.211683 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:54:31.218039 1897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:31.213893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54148 (local address 127.3.221.126:40419)
0811 11:54:31.214068 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:31.214071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:31.214188 (+ 117us) server_negotiation.cc:407] Connection header received
0811 11:54:31.214250 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:31.214252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:31.214301 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:31.214377 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:31.214663 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.215293 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.216183 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.216468 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.216563 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:31.217591 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:31.217595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:31.217597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:31.217612 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:31.217633 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:31.217641 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:31.217736 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:31.217870 (+ 134us) server_negotiation.cc:299] Negotiation successful
0811 11:54:31.217912 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 11:54:31.221479 1836 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:54:31.224664 1836 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:54:31.241560 2594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:31.234960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:40221 (local address 127.0.0.1:45618)
0811 11:54:31.235639 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:31.235654 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:31.235749 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:31.236882 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:31.236885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:31.236902 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:31.236986 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:31.236992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.238856 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.238860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:31.240035 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:31.240047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.240954 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.240958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:31.241070 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:31.241109 (+ 39us) client_negotiation.cc:769] Sending connection context
0811 11:54:31.241265 (+ 156us) client_negotiation.cc:240] Negotiation successful
0811 11:54:31.241409 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260811 11:54:31.241564 2597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:31.235934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45618 (local address 127.3.221.66:40221)
0811 11:54:31.236278 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:31.236291 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:31.236314 (+ 23us) server_negotiation.cc:407] Connection header received
0811 11:54:31.236389 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:31.236394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:31.236583 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:31.236734 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:31.237107 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.238283 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.240187 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.240835 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.240935 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:31.241159 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:31.241281 (+ 122us) server_negotiation.cc:299] Negotiation successful
0811 11:54:31.241396 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":116,"threads_started":1}
I20260811 11:54:31.242110 2600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:31.236725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:33059 (local address 127.0.0.1:49634)
0811 11:54:31.237063 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:31.237077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:31.237165 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:31.237833 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:31.237836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:31.237856 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:31.237968 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:31.237975 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.239094 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.239098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:31.240339 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:31.240348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.241696 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.241700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:31.241807 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:31.241831 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 11:54:31.241904 (+ 73us) client_negotiation.cc:240] Negotiation successful
0811 11:54:31.241982 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":109,"thread_start_us":124,"threads_started":1}
I20260811 11:54:31.242336 2602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:31.237023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49634 (local address 127.3.221.65:33059)
0811 11:54:31.237393 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:31.237406 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:31.237430 (+ 24us) server_negotiation.cc:407] Connection header received
0811 11:54:31.237507 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:31.237512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:31.237694 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:31.237853 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:31.238092 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.238965 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.240478 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.241341 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.241455 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:31.241630 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:31.241942 (+ 312us) server_negotiation.cc:299] Negotiation successful
0811 11:54:31.242047 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":153,"threads_started":1}
I20260811 11:54:31.242990 2596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:31.235801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44740 (local address 127.3.221.67:44323)
0811 11:54:31.236189 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:31.236202 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:31.236228 (+ 26us) server_negotiation.cc:407] Connection header received
0811 11:54:31.236318 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:31.236324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:31.236500 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:31.236648 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:31.237479 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.238517 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.240078 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.240864 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.240979 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:31.241169 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:31.241551 (+ 382us) server_negotiation.cc:299] Negotiation successful
0811 11:54:31.242817 (+ 1266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":155,"threads_started":1}
I20260811 11:54:31.243536 2595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:31.234889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:44323 (local address 127.0.0.1:44740)
0811 11:54:31.235450 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:31.235508 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:31.235664 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:31.236655 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:31.236661 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:31.236683 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:31.236843 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:31.236851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.238638 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.238641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:31.239944 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:31.239954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.241294 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.241297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:31.241387 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:31.241415 (+ 28us) client_negotiation.cc:769] Sending connection context
0811 11:54:31.243324 (+ 1909us) client_negotiation.cc:240] Negotiation successful
0811 11:54:31.243409 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":30,"thread_start_us":115,"threads_started":1}
I20260811 11:54:31.245083 2038 tablet_service.cc:1467] Processing CreateTablet for tablet 065628acfb1741d48a294e0a1b7ee1b8 (DEFAULT_TABLE table=test-table [id=46db599afb58444096f47701e80ba663]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:54:31.245045 2254 tablet_service.cc:1467] Processing CreateTablet for tablet 065628acfb1741d48a294e0a1b7ee1b8 (DEFAULT_TABLE table=test-table [id=46db599afb58444096f47701e80ba663]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:54:31.246176 2038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065628acfb1741d48a294e0a1b7ee1b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:31.246207 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065628acfb1741d48a294e0a1b7ee1b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:31.246521 2468 tablet_service.cc:1467] Processing CreateTablet for tablet 065628acfb1741d48a294e0a1b7ee1b8 (DEFAULT_TABLE table=test-table [id=46db599afb58444096f47701e80ba663]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:54:31.247749 2468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065628acfb1741d48a294e0a1b7ee1b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:31.253585 2603 tablet_bootstrap.cc:492] T 065628acfb1741d48a294e0a1b7ee1b8 P 2713053d46714c879a8057674ddff964: Bootstrap starting.
I20260811 11:54:31.255033 2604 tablet_bootstrap.cc:492] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba: Bootstrap starting.
I20260811 11:54:31.255918 2603 tablet_bootstrap.cc:654] T 065628acfb1741d48a294e0a1b7ee1b8 P 2713053d46714c879a8057674ddff964: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:31.256640 2603 log.cc:824] T 065628acfb1741d48a294e0a1b7ee1b8 P 2713053d46714c879a8057674ddff964: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:31.256922 2604 tablet_bootstrap.cc:654] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:31.257413 2605 tablet_bootstrap.cc:492] T 065628acfb1741d48a294e0a1b7ee1b8 P fedcee92d3bc48b1b27d6dc63820aa3d: Bootstrap starting.
I20260811 11:54:31.257815 2604 log.cc:824] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:31.258927 2603 tablet_bootstrap.cc:492] T 065628acfb1741d48a294e0a1b7ee1b8 P 2713053d46714c879a8057674ddff964: No bootstrap required, opened a new log
I20260811 11:54:31.259130 2603 ts_tablet_manager.cc:1397] T 065628acfb1741d48a294e0a1b7ee1b8 P 2713053d46714c879a8057674ddff964: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 11:54:31.259632 2605 tablet_bootstrap.cc:654] T 065628acfb1741d48a294e0a1b7ee1b8 P fedcee92d3bc48b1b27d6dc63820aa3d: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:31.259938 2604 tablet_bootstrap.cc:492] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba: No bootstrap required, opened a new log
I20260811 11:54:31.260133 2604 ts_tablet_manager.cc:1397] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260811 11:54:31.260501 2605 log.cc:824] T 065628acfb1741d48a294e0a1b7ee1b8 P fedcee92d3bc48b1b27d6dc63820aa3d: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:31.262861 2605 tablet_bootstrap.cc:492] T 065628acfb1741d48a294e0a1b7ee1b8 P fedcee92d3bc48b1b27d6dc63820aa3d: No bootstrap required, opened a new log
I20260811 11:54:31.263070 2605 ts_tablet_manager.cc:1397] T 065628acfb1741d48a294e0a1b7ee1b8 P fedcee92d3bc48b1b27d6dc63820aa3d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 11:54:31.262931 2603 raft_consensus.cc:348] T 065628acfb1741d48a294e0a1b7ee1b8 P 2713053d46714c879a8057674ddff964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedcee92d3bc48b1b27d6dc63820aa3d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44323 } } peers { permanent_uuid: "ebcb585c28d54c239bfc7b1fe71302ba" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40221 } } peers { permanent_uuid: "2713053d46714c879a8057674ddff964" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 33059 } }
I20260811 11:54:31.263223 2603 raft_consensus.cc:374] T 065628acfb1741d48a294e0a1b7ee1b8 P 2713053d46714c879a8057674ddff964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:31.263278 2603 raft_consensus.cc:729] T 065628acfb1741d48a294e0a1b7ee1b8 P 2713053d46714c879a8057674ddff964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2713053d46714c879a8057674ddff964, State: Initialized, Role: FOLLOWER
I20260811 11:54:31.263741 2603 consensus_queue.cc:260] T 065628acfb1741d48a294e0a1b7ee1b8 P 2713053d46714c879a8057674ddff964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedcee92d3bc48b1b27d6dc63820aa3d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44323 } } peers { permanent_uuid: "ebcb585c28d54c239bfc7b1fe71302ba" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40221 } } peers { permanent_uuid: "2713053d46714c879a8057674ddff964" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 33059 } }
I20260811 11:54:31.264447 2157 heartbeater.cc:503] Master 127.3.221.126:40419 was elected leader, sending a full tablet report...
I20260811 11:54:31.264832 2603 ts_tablet_manager.cc:1428] T 065628acfb1741d48a294e0a1b7ee1b8 P 2713053d46714c879a8057674ddff964: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260811 11:54:31.264431 2604 raft_consensus.cc:348] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedcee92d3bc48b1b27d6dc63820aa3d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44323 } } peers { permanent_uuid: "ebcb585c28d54c239bfc7b1fe71302ba" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40221 } } peers { permanent_uuid: "2713053d46714c879a8057674ddff964" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 33059 } }
I20260811 11:54:31.265970 2604 raft_consensus.cc:374] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:31.266045 2604 raft_consensus.cc:729] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebcb585c28d54c239bfc7b1fe71302ba, State: Initialized, Role: FOLLOWER
I20260811 11:54:31.266537 2604 consensus_queue.cc:260] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedcee92d3bc48b1b27d6dc63820aa3d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44323 } } peers { permanent_uuid: "ebcb585c28d54c239bfc7b1fe71302ba" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40221 } } peers { permanent_uuid: "2713053d46714c879a8057674ddff964" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 33059 } }
I20260811 11:54:31.267278 2370 heartbeater.cc:503] Master 127.3.221.126:40419 was elected leader, sending a full tablet report...
I20260811 11:54:31.267284 2605 raft_consensus.cc:348] T 065628acfb1741d48a294e0a1b7ee1b8 P fedcee92d3bc48b1b27d6dc63820aa3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedcee92d3bc48b1b27d6dc63820aa3d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44323 } } peers { permanent_uuid: "ebcb585c28d54c239bfc7b1fe71302ba" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40221 } } peers { permanent_uuid: "2713053d46714c879a8057674ddff964" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 33059 } }
I20260811 11:54:31.267537 2605 raft_consensus.cc:374] T 065628acfb1741d48a294e0a1b7ee1b8 P fedcee92d3bc48b1b27d6dc63820aa3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:31.267584 2605 raft_consensus.cc:729] T 065628acfb1741d48a294e0a1b7ee1b8 P fedcee92d3bc48b1b27d6dc63820aa3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fedcee92d3bc48b1b27d6dc63820aa3d, State: Initialized, Role: FOLLOWER
I20260811 11:54:31.267653 2604 ts_tablet_manager.cc:1428] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260811 11:54:31.268046 2605 consensus_queue.cc:260] T 065628acfb1741d48a294e0a1b7ee1b8 P fedcee92d3bc48b1b27d6dc63820aa3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedcee92d3bc48b1b27d6dc63820aa3d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44323 } } peers { permanent_uuid: "ebcb585c28d54c239bfc7b1fe71302ba" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40221 } } peers { permanent_uuid: "2713053d46714c879a8057674ddff964" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 33059 } }
I20260811 11:54:31.268930 2605 ts_tablet_manager.cc:1428] T 065628acfb1741d48a294e0a1b7ee1b8 P fedcee92d3bc48b1b27d6dc63820aa3d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260811 11:54:31.269307 2575 heartbeater.cc:503] Master 127.3.221.126:40419 was elected leader, sending a full tablet report...
W20260811 11:54:31.313001 2158 tablet.cc:2369] T 065628acfb1741d48a294e0a1b7ee1b8 P 2713053d46714c879a8057674ddff964: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:54:31.321651 2612 raft_consensus.cc:482] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:54:31.321867 2612 raft_consensus.cc:504] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedcee92d3bc48b1b27d6dc63820aa3d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44323 } } peers { permanent_uuid: "ebcb585c28d54c239bfc7b1fe71302ba" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40221 } } peers { permanent_uuid: "2713053d46714c879a8057674ddff964" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 33059 } }
I20260811 11:54:31.323006 2612 leader_election.cc:288] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fedcee92d3bc48b1b27d6dc63820aa3d (127.3.221.67:44323), 2713053d46714c879a8057674ddff964 (127.3.221.65:33059)
I20260811 11:54:31.328023 2602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:31.323770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:32783 (local address 127.3.221.65:33059)
0811 11:54:31.323900 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:31.323905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:31.324145 (+ 240us) server_negotiation.cc:407] Connection header received
0811 11:54:31.324226 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:31.324229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:31.324274 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:31.324356 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:31.324587 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.325136 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.326630 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.327245 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.327385 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:31.327441 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:31.327793 (+ 352us) server_negotiation.cc:299] Negotiation successful
0811 11:54:31.327877 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 11:54:31.328023 2596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:31.323450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:55249 (local address 127.3.221.67:44323)
0811 11:54:31.323585 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:31.323590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:31.323607 (+ 17us) server_negotiation.cc:407] Connection header received
0811 11:54:31.323763 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:31.323766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:31.323821 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:31.323910 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:31.324464 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.324975 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.326630 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.327245 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.327385 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:31.327441 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:31.327794 (+ 353us) server_negotiation.cc:299] Negotiation successful
0811 11:54:31.327877 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 11:54:31.328042 2372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:31.323314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:44323 (local address 127.3.221.66:55249)
0811 11:54:31.323454 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:31.323468 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:31.323587 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:31.323974 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:31.323977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:31.323999 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:31.324134 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:31.324141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.325109 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.325113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:31.326469 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:31.326477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.327481 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.327484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:31.327623 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:31.327676 (+ 53us) client_negotiation.cc:769] Sending connection context
0811 11:54:31.327766 (+ 90us) client_negotiation.cc:240] Negotiation successful
0811 11:54:31.327881 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":74}
I20260811 11:54:31.328048 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:31.323598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:33059 (local address 127.3.221.66:32783)
0811 11:54:31.324030 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:31.324043 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:31.324141 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:31.324362 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:31.324364 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:31.324377 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:31.324474 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:31.324479 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.325269 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.325272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:31.326464 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:31.326477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.327479 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.327483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:31.327647 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:31.327682 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 11:54:31.327766 (+ 84us) client_negotiation.cc:240] Negotiation successful
0811 11:54:31.327881 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":100,"thread_start_us":182,"threads_started":1}
I20260811 11:54:31.328790 2500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065628acfb1741d48a294e0a1b7ee1b8" candidate_uuid: "ebcb585c28d54c239bfc7b1fe71302ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fedcee92d3bc48b1b27d6dc63820aa3d" is_pre_election: true
I20260811 11:54:31.328896 2077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065628acfb1741d48a294e0a1b7ee1b8" candidate_uuid: "ebcb585c28d54c239bfc7b1fe71302ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2713053d46714c879a8057674ddff964" is_pre_election: true
I20260811 11:54:31.329120 2500 raft_consensus.cc:2454] T 065628acfb1741d48a294e0a1b7ee1b8 P fedcee92d3bc48b1b27d6dc63820aa3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebcb585c28d54c239bfc7b1fe71302ba in term 0.
I20260811 11:54:31.329196 2077 raft_consensus.cc:2454] T 065628acfb1741d48a294e0a1b7ee1b8 P 2713053d46714c879a8057674ddff964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebcb585c28d54c239bfc7b1fe71302ba in term 0.
I20260811 11:54:31.329643 2181 leader_election.cc:302] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ebcb585c28d54c239bfc7b1fe71302ba, fedcee92d3bc48b1b27d6dc63820aa3d; no voters:
I20260811 11:54:31.329919 2612 raft_consensus.cc:2781] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:54:31.330009 2612 raft_consensus.cc:482] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:54:31.330061 2612 raft_consensus.cc:3037] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:31.331229 2612 raft_consensus.cc:504] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedcee92d3bc48b1b27d6dc63820aa3d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44323 } } peers { permanent_uuid: "ebcb585c28d54c239bfc7b1fe71302ba" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40221 } } peers { permanent_uuid: "2713053d46714c879a8057674ddff964" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 33059 } }
I20260811 11:54:31.331594 2612 leader_election.cc:288] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba [CANDIDATE]: Term 1 election: Requested vote from peers fedcee92d3bc48b1b27d6dc63820aa3d (127.3.221.67:44323), 2713053d46714c879a8057674ddff964 (127.3.221.65:33059)
I20260811 11:54:31.332069 2077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065628acfb1741d48a294e0a1b7ee1b8" candidate_uuid: "ebcb585c28d54c239bfc7b1fe71302ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2713053d46714c879a8057674ddff964"
I20260811 11:54:31.332207 2077 raft_consensus.cc:3037] T 065628acfb1741d48a294e0a1b7ee1b8 P 2713053d46714c879a8057674ddff964 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:31.332342 2500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065628acfb1741d48a294e0a1b7ee1b8" candidate_uuid: "ebcb585c28d54c239bfc7b1fe71302ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fedcee92d3bc48b1b27d6dc63820aa3d"
I20260811 11:54:31.332628 2500 raft_consensus.cc:3037] T 065628acfb1741d48a294e0a1b7ee1b8 P fedcee92d3bc48b1b27d6dc63820aa3d [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:31.333472 2077 raft_consensus.cc:2454] T 065628acfb1741d48a294e0a1b7ee1b8 P 2713053d46714c879a8057674ddff964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebcb585c28d54c239bfc7b1fe71302ba in term 1.
I20260811 11:54:31.333941 2181 leader_election.cc:302] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2713053d46714c879a8057674ddff964, ebcb585c28d54c239bfc7b1fe71302ba; no voters:
I20260811 11:54:31.334211 2612 raft_consensus.cc:2781] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:31.334456 2612 raft_consensus.cc:686] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba [term 1 LEADER]: Becoming Leader. State: Replica: ebcb585c28d54c239bfc7b1fe71302ba, State: Running, Role: LEADER
I20260811 11:54:31.334579 2500 raft_consensus.cc:2454] T 065628acfb1741d48a294e0a1b7ee1b8 P fedcee92d3bc48b1b27d6dc63820aa3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebcb585c28d54c239bfc7b1fe71302ba in term 1.
I20260811 11:54:31.334851 2612 consensus_queue.cc:237] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedcee92d3bc48b1b27d6dc63820aa3d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44323 } } peers { permanent_uuid: "ebcb585c28d54c239bfc7b1fe71302ba" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40221 } } peers { permanent_uuid: "2713053d46714c879a8057674ddff964" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 33059 } }
I20260811 11:54:31.339089 1836 catalog_manager.cc:5495] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba reported cstate change: term changed from 0 to 1, leader changed from <none> to ebcb585c28d54c239bfc7b1fe71302ba (127.3.221.66). New cstate: current_term: 1 leader_uuid: "ebcb585c28d54c239bfc7b1fe71302ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedcee92d3bc48b1b27d6dc63820aa3d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebcb585c28d54c239bfc7b1fe71302ba" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 40221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2713053d46714c879a8057674ddff964" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 33059 } health_report { overall_health: UNKNOWN } } }
I20260811 11:54:31.368708 2597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:31.364650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.3.221.66:40221)
0811 11:54:31.364778 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:31.364782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:31.364796 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:54:31.364872 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:31.364874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:31.364933 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:31.365024 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:54:31.365203 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.365753 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.366642 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:31.366883 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:31.366975 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:31.367315 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:54:31.367860 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:54:31.367991 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:31.368356 (+ 365us) server_negotiation.cc:299] Negotiation successful
0811 11:54:31.368434 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260811 11:54:31.375080 2500 raft_consensus.cc:1261] T 065628acfb1741d48a294e0a1b7ee1b8 P fedcee92d3bc48b1b27d6dc63820aa3d [term 1 FOLLOWER]: Refusing update from remote peer ebcb585c28d54c239bfc7b1fe71302ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:54:31.375080 2077 raft_consensus.cc:1261] T 065628acfb1741d48a294e0a1b7ee1b8 P 2713053d46714c879a8057674ddff964 [term 1 FOLLOWER]: Refusing update from remote peer ebcb585c28d54c239bfc7b1fe71302ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:54:31.375829 2616 consensus_queue.cc:1035] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "fedcee92d3bc48b1b27d6dc63820aa3d" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 44323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:54:31.376132 2612 consensus_queue.cc:1035] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "2713053d46714c879a8057674ddff964" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 33059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:54:31.396037 1836 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:54:31.396276 1836 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:54:31.400168 1836 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c5017af30b8414889ab8e647f6979b4: Sending DeleteTablet for 3 replicas of tablet 065628acfb1741d48a294e0a1b7ee1b8
I20260811 11:54:31.401149 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet 065628acfb1741d48a294e0a1b7ee1b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:54:31 UTC) from {username='slave'} at 127.0.0.1:45618
I20260811 11:54:31.401237 2468 tablet_service.cc:1514] Processing DeleteTablet for tablet 065628acfb1741d48a294e0a1b7ee1b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:54:31 UTC) from {username='slave'} at 127.0.0.1:44740
I20260811 11:54:31.401824 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 1931
I20260811 11:54:31.401811 2634 tablet_replica.cc:330] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba: stopping tablet replica
I20260811 11:54:31.402195 2634 raft_consensus.cc:2229] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:54:31.402573 2634 raft_consensus.cc:2258] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:54:31.408339 2634 ts_tablet_manager.cc:1905] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:54:31.411506 2634 ts_tablet_manager.cc:1918] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:54:31.411922 2631 tablet_replica.cc:330] T 065628acfb1741d48a294e0a1b7ee1b8 P fedcee92d3bc48b1b27d6dc63820aa3d: stopping tablet replica
I20260811 11:54:31.412096 2634 log.cc:1196] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/065628acfb1741d48a294e0a1b7ee1b8
I20260811 11:54:31.412591 2634 ts_tablet_manager.cc:1939] T 065628acfb1741d48a294e0a1b7ee1b8 P ebcb585c28d54c239bfc7b1fe71302ba: Deleting consensus metadata
I20260811 11:54:31.413048 2631 raft_consensus.cc:2229] T 065628acfb1741d48a294e0a1b7ee1b8 P fedcee92d3bc48b1b27d6dc63820aa3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:54:31.413558 2631 raft_consensus.cc:2258] T 065628acfb1741d48a294e0a1b7ee1b8 P fedcee92d3bc48b1b27d6dc63820aa3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:54:31.414613 1810 catalog_manager.cc:4841] TS ebcb585c28d54c239bfc7b1fe71302ba (127.3.221.66:40221): tablet 065628acfb1741d48a294e0a1b7ee1b8 (table test-table [id=46db599afb58444096f47701e80ba663]) successfully deleted
I20260811 11:54:31.415557 2631 ts_tablet_manager.cc:1905] T 065628acfb1741d48a294e0a1b7ee1b8 P fedcee92d3bc48b1b27d6dc63820aa3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:54:31.418578 2631 ts_tablet_manager.cc:1918] T 065628acfb1741d48a294e0a1b7ee1b8 P fedcee92d3bc48b1b27d6dc63820aa3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:54:31.418690 2631 log.cc:1196] T 065628acfb1741d48a294e0a1b7ee1b8 P fedcee92d3bc48b1b27d6dc63820aa3d: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/065628acfb1741d48a294e0a1b7ee1b8
I20260811 11:54:31.419009 2631 ts_tablet_manager.cc:1939] T 065628acfb1741d48a294e0a1b7ee1b8 P fedcee92d3bc48b1b27d6dc63820aa3d: Deleting consensus metadata
I20260811 11:54:31.420513 1808 catalog_manager.cc:4841] TS fedcee92d3bc48b1b27d6dc63820aa3d (127.3.221.67:44323): tablet 065628acfb1741d48a294e0a1b7ee1b8 (table test-table [id=46db599afb58444096f47701e80ba663]) successfully deleted
W20260811 11:54:31.420758 1808 connection.cc:535] client connection to 127.3.221.65:33059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260811 11:54:31.420846 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 2160
W20260811 11:54:31.420903 1808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260811 11:54:31.425112 2595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:31.424788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:33059 (local address 127.0.0.1:49648)
0811 11:54:31.424953 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:31.425027 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.65:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260811 11:54:31.425328 1808 catalog_manager.cc:4584] TS 2713053d46714c879a8057674ddff964 (127.3.221.65:33059): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 065628acfb1741d48a294e0a1b7ee1b8: Network error: Client connection negotiation failed: client connection to 127.3.221.65:33059: connect: Connection refused (error 111)
I20260811 11:54:31.430780 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 2373
I20260811 11:54:31.449458 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 1783
2026-08-11T11:54:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1146 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:54:31 dist-test-slave-bndk krb5kdc[2640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:54:31 dist-test-slave-bndk krb5kdc[2640](info): set up 2 sockets
Aug 11 11:54:31 dist-test-slave-bndk krb5kdc[2640](info): commencing operation
krb5kdc: starting...
W20260811 11:54:32.648864 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:54:32 dist-test-slave-bndk krb5kdc[2640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:54:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:54:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:54:32.792423 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:41071
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:43785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:41071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:32.918413 2658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:32.918753 2658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:32.918815 2658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:32.923309 2658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:54:32.923388 2658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:32.923416 2658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:32.923437 2658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:54:32.923458 2658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:54:32.931298 2658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:43785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:41071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:41071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:32.932960 2658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:32.934664 2658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:32.945356 2674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:32.945344 2671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:32.945344 2669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:32.945777 2658 server_base.cc:1034] running on GCE node
I20260811 11:54:32.946470 2658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:32.947775 2658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:32.948989 2658 hybrid_clock.cc:648] HybridClock initialized: now 1786449272948962 us; error 44 us; skew 500 ppm
Aug 11 11:54:32 dist-test-slave-bndk krb5kdc[2640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449272, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:32.952185 2658 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:54:32.953265 2658 webserver.cc:460] Webserver started at http://127.3.221.126:44671/ using document root <none> and password file <none>
I20260811 11:54:32.953881 2658 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:32.953953 2658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:32.954154 2658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:32.956395 2658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "8f43ca40d79b480bb381014c35ae9946"
format_stamp: "Formatted at 2026-08-11 11:54:32 on dist-test-slave-bndk"
server_key: "0912b1687cf6f49189ef354adc479318"
server_key_iv: "2bb319d860ae06ca9403a6ac52ab0072"
server_key_version: "encryptionkey@0"
I20260811 11:54:32.956982 2658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "8f43ca40d79b480bb381014c35ae9946"
format_stamp: "Formatted at 2026-08-11 11:54:32 on dist-test-slave-bndk"
server_key: "0912b1687cf6f49189ef354adc479318"
server_key_iv: "2bb319d860ae06ca9403a6ac52ab0072"
server_key_version: "encryptionkey@0"
I20260811 11:54:32.961156 2658 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 11:54:32.964251 2691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:32.966199 2658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:54:32.966382 2658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "8f43ca40d79b480bb381014c35ae9946"
format_stamp: "Formatted at 2026-08-11 11:54:32 on dist-test-slave-bndk"
server_key: "0912b1687cf6f49189ef354adc479318"
server_key_iv: "2bb319d860ae06ca9403a6ac52ab0072"
server_key_version: "encryptionkey@0"
I20260811 11:54:32.966517 2658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:33.001453 2658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:33.005016 2658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:33.005252 2658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:33.013033 2658 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:41071
I20260811 11:54:33.013051 2770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:41071 every 8 connection(s)
I20260811 11:54:33.014123 2658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:54:33.017099 2772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:33.022716 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 2658
I20260811 11:54:33.022862 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:54:33.023164 3957 external_mini_cluster.cc:1428] Setting key 23389b4256dcdebba3c51f60f66db932
I20260811 11:54:33.023567 2772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946: Bootstrap starting.
I20260811 11:54:33.026320 2772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:33.027109 2772 log.cc:824] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:33.029163 2772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946: No bootstrap required, opened a new log
Aug 11 11:54:33 dist-test-slave-bndk krb5kdc[2640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:54:33.033351 2772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f43ca40d79b480bb381014c35ae9946" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 41071 } }
I20260811 11:54:33.033638 2772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:33.033710 2772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f43ca40d79b480bb381014c35ae9946, State: Initialized, Role: FOLLOWER
I20260811 11:54:33.034283 2772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f43ca40d79b480bb381014c35ae9946" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 41071 } }
I20260811 11:54:33.034582 2772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:54:33.034672 2772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:54:33.034801 2772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:33.036203 2772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f43ca40d79b480bb381014c35ae9946" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 41071 } }
I20260811 11:54:33.036616 2772 leader_election.cc:302] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f43ca40d79b480bb381014c35ae9946; no voters:
I20260811 11:54:33.037034 2772 leader_election.cc:288] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:54:33.037171 2780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:33.037436 2780 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946 [term 1 LEADER]: Becoming Leader. State: Replica: 8f43ca40d79b480bb381014c35ae9946, State: Running, Role: LEADER
I20260811 11:54:33.037871 2780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f43ca40d79b480bb381014c35ae9946" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 41071 } }
I20260811 11:54:33.038427 2772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:54:33.039548 2780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f43ca40d79b480bb381014c35ae9946. Latest consensus state: current_term: 1 leader_uuid: "8f43ca40d79b480bb381014c35ae9946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f43ca40d79b480bb381014c35ae9946" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 41071 } } }
I20260811 11:54:33.039788 2780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:33.039774 2781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f43ca40d79b480bb381014c35ae9946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f43ca40d79b480bb381014c35ae9946" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 41071 } } }
I20260811 11:54:33.039902 2781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:33.040261 2791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:54:33.040810 2778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:33.024985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37176 (local address 127.3.221.126:41071)
0811 11:54:33.025534 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:33.025551 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:33.025590 (+ 39us) server_negotiation.cc:407] Connection header received
0811 11:54:33.026288 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:33.026306 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:33.026720 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:33.027070 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:33.027932 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.028965 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.029895 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.030402 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.030572 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:33.033771 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:33.033779 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:33.033799 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:33.033842 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:33.036096 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:33.036746 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:33.036753 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:33.036760 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:33.036922 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:33.037341 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:33.037344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:33.037346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:33.037978 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:33.038860 (+ 882us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:33.039034 (+ 174us) server_negotiation.cc:299] Negotiation successful
0811 11:54:33.039373 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":179,"threads_started":1}
I20260811 11:54:33.044339 2791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:54:33.050822 2791 catalog_manager.cc:1347] Generated new cluster ID: ff13888069c14c7db9d12ccf188996ae
I20260811 11:54:33.050920 2791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:54:33.064961 2791 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:54:33.066437 2791 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:54:33.074368 2791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946: Generated new TSK 0
I20260811 11:54:33.075333 2791 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
Entry for principal kudu/127.3.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab.
I20260811 11:54:33.117385 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:41071
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:43785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:33.241344 2811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:33.241655 2811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:33.241719 2811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:33.246018 2811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:33.246095 2811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:33.246182 2811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:54:33.251829 2811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:43785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.65.keytab
--principal=kudu/127.3.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:41071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:33.253078 2811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:33.254189 2811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:33.264248 2822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:33.264250 2818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:33.264250 2819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:33.264711 2811 server_base.cc:1034] running on GCE node
I20260811 11:54:33.265130 2811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:33.265838 2811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:33.267031 2811 hybrid_clock.cc:648] HybridClock initialized: now 1786449273267002 us; error 43 us; skew 500 ppm
Aug 11 11:54:33 dist-test-slave-bndk krb5kdc[2640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449273, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:33.270066 2811 init.cc:377] Logged in from keytab as kudu/127.3.221.65@KRBTEST.COM (short username kudu)
I20260811 11:54:33.271169 2811 webserver.cc:460] Webserver started at http://127.3.221.65:44927/ using document root <none> and password file <none>
I20260811 11:54:33.271785 2811 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:33.271859 2811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:33.272070 2811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:33.274273 2811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "5e5cdb3dba704dc88060df99509839a1"
format_stamp: "Formatted at 2026-08-11 11:54:33 on dist-test-slave-bndk"
server_key: "baca50d96b6b7911a193b00bae40d642"
server_key_iv: "b8625ac1a58cc2bcb4b013fd2066771f"
server_key_version: "encryptionkey@0"
I20260811 11:54:33.274845 2811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "5e5cdb3dba704dc88060df99509839a1"
format_stamp: "Formatted at 2026-08-11 11:54:33 on dist-test-slave-bndk"
server_key: "baca50d96b6b7911a193b00bae40d642"
server_key_iv: "b8625ac1a58cc2bcb4b013fd2066771f"
server_key_version: "encryptionkey@0"
I20260811 11:54:33.278925 2811 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:54:33.281682 2833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:33.282985 2811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:54:33.283113 2811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "5e5cdb3dba704dc88060df99509839a1"
format_stamp: "Formatted at 2026-08-11 11:54:33 on dist-test-slave-bndk"
server_key: "baca50d96b6b7911a193b00bae40d642"
server_key_iv: "b8625ac1a58cc2bcb4b013fd2066771f"
server_key_version: "encryptionkey@0"
I20260811 11:54:33.283229 2811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:33.303552 2811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:33.307080 2811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:33.307276 2811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:33.307927 2811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:54:33.308835 2811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:54:33.308887 2811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:33.308929 2811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:54:33.309010 2811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:33.319887 2811 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:36713
I20260811 11:54:33.319916 3007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:36713 every 8 connection(s)
I20260811 11:54:33.320914 2811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:54:33.327196 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 2811
I20260811 11:54:33.327459 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:54:33.327852 3957 external_mini_cluster.cc:1428] Setting key 90e07af34141533b8bb99a21846afc68
Aug 11 11:54:33 dist-test-slave-bndk krb5kdc[2640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449273, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.65@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:54:33.339589 2778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:33.322999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:38681 (local address 127.3.221.126:41071)
0811 11:54:33.323176 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:33.323180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:33.324094 (+ 914us) server_negotiation.cc:407] Connection header received
0811 11:54:33.325568 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:33.325571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:33.325632 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:33.325710 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:33.327765 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.328410 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.329911 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.330153 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.330246 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:33.334318 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:33.334325 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:33.334329 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:33.334399 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:33.336219 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:33.337194 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:33.337197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:33.337199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:33.337369 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:33.337973 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:33.337978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:33.337979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:33.338311 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:33.338426 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:33.339266 (+ 840us) server_negotiation.cc:299] Negotiation successful
0811 11:54:33.339447 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
WARNING: no policy specified for kudu/127.3.221.66@KRBTEST.COM; defaulting to no policy
I20260811 11:54:33.341408 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:33.323395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:41071 (local address 127.3.221.65:38681)
0811 11:54:33.323922 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:33.323971 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:33.325336 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:33.325906 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:33.325913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:33.326315 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:54:33.327572 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:33.327599 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.328564 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.328567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:33.329729 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:33.329743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.330337 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.330341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:33.330539 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:33.331360 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:54:33.331383 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:54:33.333991 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:54:33.336434 (+ 2443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:54:33.336443 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:54:33.336466 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:54:33.336991 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:54:33.337484 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:54:33.337488 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:54:33.337490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:54:33.337744 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:54:33.338457 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:54:33.338466 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:54:33.338930 (+ 464us) client_negotiation.cc:769] Sending connection context
0811 11:54:33.339182 (+ 252us) client_negotiation.cc:240] Negotiation successful
0811 11:54:33.339979 (+ 797us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":125,"threads_started":1}
I20260811 11:54:33.342893 3008 heartbeater.cc:347] Connected to a master server at 127.3.221.126:41071
I20260811 11:54:33.343204 3008 heartbeater.cc:464] Registering TS with master...
I20260811 11:54:33.343864 3008 heartbeater.cc:511] Master 127.3.221.126:41071 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.66@KRBTEST.COM" created.
I20260811 11:54:33.345767 2717 ts_manager.cc:194] Registered new tserver with Master: 5e5cdb3dba704dc88060df99509839a1 (127.3.221.65:36713)
I20260811 11:54:33.346951 2717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.65@KRBTEST.COM'} at 127.3.221.65:38681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
Entry for principal kudu/127.3.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab.
I20260811 11:54:33.360791 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:41071
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:43785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:33.482029 3016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:33.482352 3016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:33.482398 3016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:33.486799 3016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:33.486899 3016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:33.486964 3016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:54:33.493155 3016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:43785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.66.keytab
--principal=kudu/127.3.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:41071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:33.494484 3016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:33.495651 3016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:54:33.506762 3016 server_base.cc:1034] running on GCE node
W20260811 11:54:33.506701 3034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:33.506701 3027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:33.506704 3028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:33.507406 3016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:33.508235 3016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:33.509397 3016 hybrid_clock.cc:648] HybridClock initialized: now 1786449273509346 us; error 64 us; skew 500 ppm
Aug 11 11:54:33 dist-test-slave-bndk krb5kdc[2640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449273, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:33.513507 3016 init.cc:377] Logged in from keytab as kudu/127.3.221.66@KRBTEST.COM (short username kudu)
I20260811 11:54:33.514766 3016 webserver.cc:460] Webserver started at http://127.3.221.66:40717/ using document root <none> and password file <none>
I20260811 11:54:33.515686 3016 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:33.515792 3016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:33.516076 3016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:33.519315 3016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "c536ac7380ea4a75bccf2363c79014e0"
format_stamp: "Formatted at 2026-08-11 11:54:33 on dist-test-slave-bndk"
server_key: "c13edacd34d9f45d95968d8d4dec6f79"
server_key_iv: "e0bc7145cdd285968d6a809e699c6bcc"
server_key_version: "encryptionkey@0"
I20260811 11:54:33.520094 3016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "c536ac7380ea4a75bccf2363c79014e0"
format_stamp: "Formatted at 2026-08-11 11:54:33 on dist-test-slave-bndk"
server_key: "c13edacd34d9f45d95968d8d4dec6f79"
server_key_iv: "e0bc7145cdd285968d6a809e699c6bcc"
server_key_version: "encryptionkey@0"
I20260811 11:54:33.524576 3016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260811 11:54:33.527391 3045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:33.528925 3016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 11:54:33.529065 3016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "c536ac7380ea4a75bccf2363c79014e0"
format_stamp: "Formatted at 2026-08-11 11:54:33 on dist-test-slave-bndk"
server_key: "c13edacd34d9f45d95968d8d4dec6f79"
server_key_iv: "e0bc7145cdd285968d6a809e699c6bcc"
server_key_version: "encryptionkey@0"
I20260811 11:54:33.529194 3016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:33.557570 3016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:33.560801 3016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:33.560995 3016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:33.561676 3016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:54:33.562575 3016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:54:33.562628 3016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:33.562683 3016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:54:33.562711 3016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:33.572278 3016 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:35343
I20260811 11:54:33.572331 3217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:35343 every 8 connection(s)
I20260811 11:54:33.573269 3016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:54:33.579803 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 3016
I20260811 11:54:33.579953 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:54:33.580235 3957 external_mini_cluster.cc:1428] Setting key eb14f0e71ef3de77bfbca7a767c64553
Aug 11 11:54:33 dist-test-slave-bndk krb5kdc[2640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449273, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.66@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:54:33.589620 2778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:33.575657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:50975 (local address 127.3.221.126:41071)
0811 11:54:33.575782 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:33.575798 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:33.576954 (+ 1156us) server_negotiation.cc:407] Connection header received
0811 11:54:33.578390 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:33.578394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:33.578456 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:33.578531 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:33.579968 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.580601 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.581950 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.582171 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.582251 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:33.585339 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:33.585344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:33.585346 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:33.585377 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:33.587066 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:33.587752 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:33.587755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:33.587757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:33.587852 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:33.588231 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:33.588234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:33.588235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:33.588464 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:33.588544 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:33.589359 (+ 815us) server_negotiation.cc:299] Negotiation successful
0811 11:54:33.589486 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 11:54:33.590830 3221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:33.576105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:41071 (local address 127.3.221.66:50975)
0811 11:54:33.576785 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:33.576831 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:33.578157 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:33.578723 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:33.578730 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:33.579121 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:54:33.579759 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:33.579782 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.580737 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.580740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:33.581828 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:33.581841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.582332 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.582336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:33.582528 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:33.583181 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:54:33.583202 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:54:33.585126 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:54:33.587223 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:54:33.587229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:54:33.587244 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:54:33.587621 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:54:33.587939 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:54:33.587942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:54:33.587943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:54:33.588114 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:54:33.588570 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:54:33.588578 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:54:33.588981 (+ 403us) client_negotiation.cc:769] Sending connection context
0811 11:54:33.589250 (+ 269us) client_negotiation.cc:240] Negotiation successful
0811 11:54:33.589672 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.3.221.67@KRBTEST.COM; defaulting to no policy
I20260811 11:54:33.592432 3218 heartbeater.cc:347] Connected to a master server at 127.3.221.126:41071
I20260811 11:54:33.592710 3218 heartbeater.cc:464] Registering TS with master...
I20260811 11:54:33.593360 3218 heartbeater.cc:511] Master 127.3.221.126:41071 requested a full tablet report, sending...
I20260811 11:54:33.594655 2717 ts_manager.cc:194] Registered new tserver with Master: c536ac7380ea4a75bccf2363c79014e0 (127.3.221.66:35343)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.67@KRBTEST.COM" created.
I20260811 11:54:33.595359 2717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.66@KRBTEST.COM'} at 127.3.221.66:50975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
Entry for principal kudu/127.3.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab.
I20260811 11:54:33.611742 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:41071
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:43785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:33.736402 3228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:33.736716 3228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:33.736763 3228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:33.741498 3228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:33.741681 3228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:33.741837 3228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:54:33.748538 3228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:43785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.67.keytab
--principal=kudu/127.3.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:41071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:33.749827 3228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:33.750892 3228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:33.762341 3238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:33.762312 3243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:33.762470 3228 server_base.cc:1034] running on GCE node
W20260811 11:54:33.762312 3239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:33.763044 3228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:33.763780 3228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:33.764984 3228 hybrid_clock.cc:648] HybridClock initialized: now 1786449273764963 us; error 34 us; skew 500 ppm
Aug 11 11:54:33 dist-test-slave-bndk krb5kdc[2640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449273, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:33.768366 3228 init.cc:377] Logged in from keytab as kudu/127.3.221.67@KRBTEST.COM (short username kudu)
I20260811 11:54:33.769516 3228 webserver.cc:460] Webserver started at http://127.3.221.67:40131/ using document root <none> and password file <none>
I20260811 11:54:33.770406 3228 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:33.770507 3228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:33.770748 3228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:33.772994 3228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "6becc20b11b34ef8ae3651a519b8ff8c"
format_stamp: "Formatted at 2026-08-11 11:54:33 on dist-test-slave-bndk"
server_key: "1efa1f9f397d9668b6b434e8fdd0b4bb"
server_key_iv: "51195be074fe57e13a4c586d4b18c617"
server_key_version: "encryptionkey@0"
I20260811 11:54:33.773780 3228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "6becc20b11b34ef8ae3651a519b8ff8c"
format_stamp: "Formatted at 2026-08-11 11:54:33 on dist-test-slave-bndk"
server_key: "1efa1f9f397d9668b6b434e8fdd0b4bb"
server_key_iv: "51195be074fe57e13a4c586d4b18c617"
server_key_version: "encryptionkey@0"
I20260811 11:54:33.778553 3228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:54:33.781942 3258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:33.783433 3228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 11:54:33.783584 3228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "6becc20b11b34ef8ae3651a519b8ff8c"
format_stamp: "Formatted at 2026-08-11 11:54:33 on dist-test-slave-bndk"
server_key: "1efa1f9f397d9668b6b434e8fdd0b4bb"
server_key_iv: "51195be074fe57e13a4c586d4b18c617"
server_key_version: "encryptionkey@0"
I20260811 11:54:33.783705 3228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:33.807185 3228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:33.810645 3228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:33.810855 3228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:33.811513 3228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:54:33.812407 3228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:54:33.812456 3228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:33.812513 3228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:54:33.812542 3228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:33.821731 3228 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:34091
I20260811 11:54:33.821751 3434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:34091 every 8 connection(s)
I20260811 11:54:33.822727 3228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:54:33.830327 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 3228
I20260811 11:54:33.830477 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:54:33.830822 3957 external_mini_cluster.cc:1428] Setting key 34d035b51357bc429c9e1ec2d7fa9e91
Aug 11 11:54:33 dist-test-slave-bndk krb5kdc[2640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449273, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.67@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:54:33.838789 2778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:33.824975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:43383 (local address 127.3.221.126:41071)
0811 11:54:33.825121 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:33.825125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:33.826097 (+ 972us) server_negotiation.cc:407] Connection header received
0811 11:54:33.827692 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:33.827696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:33.827774 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:33.827848 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:33.829175 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.829761 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.831037 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.831355 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.831613 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:33.835012 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:33.835015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:33.835017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:33.835046 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:33.836547 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:33.837123 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:33.837126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:33.837127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:33.837209 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:33.837577 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:33.837580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:33.837581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:33.837795 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:33.837884 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:33.838469 (+ 585us) server_negotiation.cc:299] Negotiation successful
0811 11:54:33.838605 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 11:54:33.839720 3441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:33.825360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:41071 (local address 127.3.221.67:43383)
0811 11:54:33.825914 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:33.825962 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:33.827429 (+ 1467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:33.828061 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:33.828069 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:33.828490 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:54:33.829015 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:33.829030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.829892 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.829895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:33.830899 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:33.830910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.831653 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.831657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:33.831861 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:33.832481 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:54:33.832502 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:54:33.834803 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:54:33.836702 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:54:33.836709 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:54:33.836727 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:54:33.837018 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:54:33.837325 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:54:33.837328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:54:33.837330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:54:33.837482 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:54:33.837922 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:54:33.837931 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:54:33.838213 (+ 282us) client_negotiation.cc:769] Sending connection context
0811 11:54:33.838410 (+ 197us) client_negotiation.cc:240] Negotiation successful
0811 11:54:33.838802 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":142,"threads_started":1}
I20260811 11:54:33.841099 3437 heartbeater.cc:347] Connected to a master server at 127.3.221.126:41071
I20260811 11:54:33.841380 3437 heartbeater.cc:464] Registering TS with master...
I20260811 11:54:33.841938 3437 heartbeater.cc:511] Master 127.3.221.126:41071 requested a full tablet report, sending...
I20260811 11:54:33.843076 2717 ts_manager.cc:194] Registered new tserver with Master: 6becc20b11b34ef8ae3651a519b8ff8c (127.3.221.67:34091)
I20260811 11:54:33.843684 2717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.221.67@KRBTEST.COM'} at 127.3.221.67:43383
I20260811 11:54:33.845621 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:54:33.855327 2778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:33.847686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37186 (local address 127.3.221.126:41071)
0811 11:54:33.847856 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:33.847861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:33.848002 (+ 141us) server_negotiation.cc:407] Connection header received
0811 11:54:33.848105 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:33.848107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:33.848158 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:33.848225 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:33.848997 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.849584 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.850464 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.850692 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.850792 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:33.852237 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:33.852240 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:33.852242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:33.852269 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:33.853766 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:33.854189 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:33.854191 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:33.854192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:33.854263 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:33.854473 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:33.854475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:33.854476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:33.854698 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:33.854799 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:33.855055 (+ 256us) server_negotiation.cc:299] Negotiation successful
0811 11:54:33.855155 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 11:54:33.858665 2717 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:54:33.861883 2717 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:54:33.877442 3455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:33.871359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:35343 (local address 127.0.0.1:56140)
0811 11:54:33.871823 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:33.871866 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:33.871996 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:33.872812 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:33.872819 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:33.872847 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:33.872980 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:33.872986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.874422 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.874425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:33.875638 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:33.875651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.876795 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.876798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:33.876921 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:33.876972 (+ 51us) client_negotiation.cc:769] Sending connection context
0811 11:54:33.877095 (+ 123us) client_negotiation.cc:240] Negotiation successful
0811 11:54:33.877267 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":23,"thread_start_us":128,"threads_started":1}
I20260811 11:54:33.877455 3457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:33.871498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56140 (local address 127.3.221.66:35343)
0811 11:54:33.872275 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:33.872288 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:33.872313 (+ 25us) server_negotiation.cc:407] Connection header received
0811 11:54:33.872395 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:33.872400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:33.872624 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:33.873127 (+ 503us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:33.873326 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.874310 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.875810 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.876666 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.876774 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:33.876938 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:33.877132 (+ 194us) server_negotiation.cc:299] Negotiation successful
0811 11:54:33.877279 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":118,"threads_started":1}
I20260811 11:54:33.878377 3456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:33.872193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:34091 (local address 127.0.0.1:39210)
0811 11:54:33.873149 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:33.873161 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:33.873354 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:33.873839 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:33.873842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:33.873859 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:33.873936 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:33.873941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.875427 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.875431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:33.876686 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:33.876698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.877871 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.877874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:33.878004 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:33.878037 (+ 33us) client_negotiation.cc:769] Sending connection context
0811 11:54:33.878114 (+ 77us) client_negotiation.cc:240] Negotiation successful
0811 11:54:33.878218 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"mutex_wait_us":43,"thread_start_us":111,"threads_started":1}
I20260811 11:54:33.878387 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:33.872375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39210 (local address 127.3.221.67:34091)
0811 11:54:33.872807 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:33.872819 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:33.873275 (+ 456us) server_negotiation.cc:407] Connection header received
0811 11:54:33.873524 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:33.873529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:33.873705 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:33.873860 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:33.874046 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.875208 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.876869 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.877751 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.877861 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:33.877996 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:33.878148 (+ 152us) server_negotiation.cc:299] Negotiation successful
0811 11:54:33.878248 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":134,"threads_started":1}
I20260811 11:54:33.878376 3458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:33.871451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:36713 (local address 127.0.0.1:60256)
0811 11:54:33.872652 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:33.872667 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:33.872756 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:33.873458 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:33.873461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:33.873480 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:33.873570 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:33.873577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.875272 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.875276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:33.876487 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:33.876500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.877839 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.877843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:33.878021 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:33.878038 (+ 17us) client_negotiation.cc:769] Sending connection context
0811 11:54:33.878114 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 11:54:33.878212 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1137,"mutex_wait_us":88,"thread_start_us":104,"threads_started":1}
I20260811 11:54:33.878840 3460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:33.872296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60256 (local address 127.3.221.65:36713)
0811 11:54:33.872973 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:33.872979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:33.873002 (+ 23us) server_negotiation.cc:407] Connection header received
0811 11:54:33.873080 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:33.873086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:33.873315 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:33.873469 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:33.873699 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.874995 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.876660 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.877498 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.877625 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:33.877768 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:33.878589 (+ 821us) server_negotiation.cc:299] Negotiation successful
0811 11:54:33.878694 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":100,"threads_started":1}
I20260811 11:54:33.881932 3329 tablet_service.cc:1467] Processing CreateTablet for tablet 05e6a9acdab84bbd9d5e83b3ee8a51ec (DEFAULT_TABLE table=test-table [id=64ab0895857f4ff987d12b5176e7c72c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:54:33.882000 3116 tablet_service.cc:1467] Processing CreateTablet for tablet 05e6a9acdab84bbd9d5e83b3ee8a51ec (DEFAULT_TABLE table=test-table [id=64ab0895857f4ff987d12b5176e7c72c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:54:33.882395 2910 tablet_service.cc:1467] Processing CreateTablet for tablet 05e6a9acdab84bbd9d5e83b3ee8a51ec (DEFAULT_TABLE table=test-table [id=64ab0895857f4ff987d12b5176e7c72c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:54:33.883478 2910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e6a9acdab84bbd9d5e83b3ee8a51ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:33.883478 3116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e6a9acdab84bbd9d5e83b3ee8a51ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:33.883585 3329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e6a9acdab84bbd9d5e83b3ee8a51ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:33.891230 3462 tablet_bootstrap.cc:492] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1: Bootstrap starting.
I20260811 11:54:33.892709 3466 tablet_bootstrap.cc:492] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P c536ac7380ea4a75bccf2363c79014e0: Bootstrap starting.
I20260811 11:54:33.893632 3465 tablet_bootstrap.cc:492] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 6becc20b11b34ef8ae3651a519b8ff8c: Bootstrap starting.
I20260811 11:54:33.893810 3462 tablet_bootstrap.cc:654] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:33.895553 3466 tablet_bootstrap.cc:654] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P c536ac7380ea4a75bccf2363c79014e0: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:33.895532 3462 log.cc:824] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:33.895874 3465 tablet_bootstrap.cc:654] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 6becc20b11b34ef8ae3651a519b8ff8c: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:33.896605 3466 log.cc:824] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P c536ac7380ea4a75bccf2363c79014e0: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:33.897127 3465 log.cc:824] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 6becc20b11b34ef8ae3651a519b8ff8c: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:33.898159 3462 tablet_bootstrap.cc:492] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1: No bootstrap required, opened a new log
I20260811 11:54:33.898376 3462 ts_tablet_manager.cc:1397] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260811 11:54:33.898802 3466 tablet_bootstrap.cc:492] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P c536ac7380ea4a75bccf2363c79014e0: No bootstrap required, opened a new log
I20260811 11:54:33.899011 3466 ts_tablet_manager.cc:1397] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P c536ac7380ea4a75bccf2363c79014e0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 11:54:33.899277 3465 tablet_bootstrap.cc:492] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 6becc20b11b34ef8ae3651a519b8ff8c: No bootstrap required, opened a new log
I20260811 11:54:33.899461 3465 ts_tablet_manager.cc:1397] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 6becc20b11b34ef8ae3651a519b8ff8c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 11:54:33.902202 3462 raft_consensus.cc:348] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c536ac7380ea4a75bccf2363c79014e0" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 35343 } } peers { permanent_uuid: "5e5cdb3dba704dc88060df99509839a1" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 36713 } } peers { permanent_uuid: "6becc20b11b34ef8ae3651a519b8ff8c" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 34091 } }
I20260811 11:54:33.902551 3462 raft_consensus.cc:374] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:33.902627 3462 raft_consensus.cc:729] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5cdb3dba704dc88060df99509839a1, State: Initialized, Role: FOLLOWER
I20260811 11:54:33.902937 3466 raft_consensus.cc:348] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P c536ac7380ea4a75bccf2363c79014e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c536ac7380ea4a75bccf2363c79014e0" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 35343 } } peers { permanent_uuid: "5e5cdb3dba704dc88060df99509839a1" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 36713 } } peers { permanent_uuid: "6becc20b11b34ef8ae3651a519b8ff8c" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 34091 } }
I20260811 11:54:33.903301 3466 raft_consensus.cc:374] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P c536ac7380ea4a75bccf2363c79014e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:33.903149 3465 raft_consensus.cc:348] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 6becc20b11b34ef8ae3651a519b8ff8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c536ac7380ea4a75bccf2363c79014e0" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 35343 } } peers { permanent_uuid: "5e5cdb3dba704dc88060df99509839a1" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 36713 } } peers { permanent_uuid: "6becc20b11b34ef8ae3651a519b8ff8c" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 34091 } }
I20260811 11:54:33.903373 3466 raft_consensus.cc:729] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P c536ac7380ea4a75bccf2363c79014e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c536ac7380ea4a75bccf2363c79014e0, State: Initialized, Role: FOLLOWER
I20260811 11:54:33.903403 3465 raft_consensus.cc:374] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 6becc20b11b34ef8ae3651a519b8ff8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:33.903342 3462 consensus_queue.cc:260] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c536ac7380ea4a75bccf2363c79014e0" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 35343 } } peers { permanent_uuid: "5e5cdb3dba704dc88060df99509839a1" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 36713 } } peers { permanent_uuid: "6becc20b11b34ef8ae3651a519b8ff8c" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 34091 } }
I20260811 11:54:33.903506 3465 raft_consensus.cc:729] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 6becc20b11b34ef8ae3651a519b8ff8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6becc20b11b34ef8ae3651a519b8ff8c, State: Initialized, Role: FOLLOWER
I20260811 11:54:33.903849 3465 consensus_queue.cc:260] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 6becc20b11b34ef8ae3651a519b8ff8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c536ac7380ea4a75bccf2363c79014e0" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 35343 } } peers { permanent_uuid: "5e5cdb3dba704dc88060df99509839a1" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 36713 } } peers { permanent_uuid: "6becc20b11b34ef8ae3651a519b8ff8c" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 34091 } }
I20260811 11:54:33.903842 3466 consensus_queue.cc:260] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P c536ac7380ea4a75bccf2363c79014e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c536ac7380ea4a75bccf2363c79014e0" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 35343 } } peers { permanent_uuid: "5e5cdb3dba704dc88060df99509839a1" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 36713 } } peers { permanent_uuid: "6becc20b11b34ef8ae3651a519b8ff8c" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 34091 } }
I20260811 11:54:33.904426 3462 ts_tablet_manager.cc:1428] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 11:54:33.904569 3437 heartbeater.cc:503] Master 127.3.221.126:41071 was elected leader, sending a full tablet report...
I20260811 11:54:33.904816 3465 ts_tablet_manager.cc:1428] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 6becc20b11b34ef8ae3651a519b8ff8c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260811 11:54:33.904817 3008 heartbeater.cc:503] Master 127.3.221.126:41071 was elected leader, sending a full tablet report...
I20260811 11:54:33.905453 3466 ts_tablet_manager.cc:1428] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P c536ac7380ea4a75bccf2363c79014e0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260811 11:54:33.905544 3218 heartbeater.cc:503] Master 127.3.221.126:41071 was elected leader, sending a full tablet report...
I20260811 11:54:33.940071 3471 raft_consensus.cc:482] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:54:33.940368 3471 raft_consensus.cc:504] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c536ac7380ea4a75bccf2363c79014e0" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 35343 } } peers { permanent_uuid: "5e5cdb3dba704dc88060df99509839a1" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 36713 } } peers { permanent_uuid: "6becc20b11b34ef8ae3651a519b8ff8c" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 34091 } }
I20260811 11:54:33.941547 3471 leader_election.cc:288] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c536ac7380ea4a75bccf2363c79014e0 (127.3.221.66:35343), 6becc20b11b34ef8ae3651a519b8ff8c (127.3.221.67:34091)
I20260811 11:54:33.945485 3457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:33.941974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:60927 (local address 127.3.221.66:35343)
0811 11:54:33.942126 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:33.942130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:33.942307 (+ 177us) server_negotiation.cc:407] Connection header received
0811 11:54:33.942465 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:33.942468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:33.942525 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:33.942610 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:33.942926 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.943425 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.944474 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.944892 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.945009 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:33.945061 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:33.945304 (+ 243us) server_negotiation.cc:299] Negotiation successful
0811 11:54:33.945368 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 11:54:33.945530 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:33.941852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:35343 (local address 127.3.221.65:60927)
0811 11:54:33.942210 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:33.942226 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:33.942352 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:33.942653 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:33.942656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:33.942690 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:33.942801 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:33.942807 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.943531 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.943534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:33.944336 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:33.944345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.945035 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.945038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:33.945141 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:33.945168 (+ 27us) client_negotiation.cc:769] Sending connection context
0811 11:54:33.945291 (+ 123us) client_negotiation.cc:240] Negotiation successful
0811 11:54:33.945367 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":135,"threads_started":1}
I20260811 11:54:33.945931 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:33.942463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:58521 (local address 127.3.221.67:34091)
0811 11:54:33.942607 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:33.942611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:33.942880 (+ 269us) server_negotiation.cc:407] Connection header received
0811 11:54:33.942991 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:33.942994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:33.943034 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:33.943094 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:33.943348 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.943827 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.944811 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.945405 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.945518 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:33.945567 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:33.945775 (+ 208us) server_negotiation.cc:299] Negotiation successful
0811 11:54:33.945840 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 11:54:33.945948 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:33.942324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:34091 (local address 127.3.221.65:58521)
0811 11:54:33.942785 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:33.942797 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:33.942892 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:33.943112 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:33.943114 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:33.943132 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:33.943243 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:33.943249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.943927 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.943931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:33.944675 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:33.944684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.945528 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.945531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:33.945629 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:33.945654 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 11:54:33.945743 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 11:54:33.945812 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":101,"threads_started":1}
I20260811 11:54:33.946406 3145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e6a9acdab84bbd9d5e83b3ee8a51ec" candidate_uuid: "5e5cdb3dba704dc88060df99509839a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c536ac7380ea4a75bccf2363c79014e0" is_pre_election: true
I20260811 11:54:33.946727 3145 raft_consensus.cc:2454] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P c536ac7380ea4a75bccf2363c79014e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5cdb3dba704dc88060df99509839a1 in term 0.
I20260811 11:54:33.947288 2838 leader_election.cc:302] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5cdb3dba704dc88060df99509839a1, c536ac7380ea4a75bccf2363c79014e0; no voters:
I20260811 11:54:33.947588 3471 raft_consensus.cc:2781] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:54:33.947602 3359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e6a9acdab84bbd9d5e83b3ee8a51ec" candidate_uuid: "5e5cdb3dba704dc88060df99509839a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6becc20b11b34ef8ae3651a519b8ff8c" is_pre_election: true
I20260811 11:54:33.947708 3471 raft_consensus.cc:482] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:54:33.947754 3471 raft_consensus.cc:3037] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:33.947898 3359 raft_consensus.cc:2454] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 6becc20b11b34ef8ae3651a519b8ff8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5cdb3dba704dc88060df99509839a1 in term 0.
I20260811 11:54:33.948904 3471 raft_consensus.cc:504] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c536ac7380ea4a75bccf2363c79014e0" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 35343 } } peers { permanent_uuid: "5e5cdb3dba704dc88060df99509839a1" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 36713 } } peers { permanent_uuid: "6becc20b11b34ef8ae3651a519b8ff8c" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 34091 } }
I20260811 11:54:33.949252 3471 leader_election.cc:288] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1 [CANDIDATE]: Term 1 election: Requested vote from peers c536ac7380ea4a75bccf2363c79014e0 (127.3.221.66:35343), 6becc20b11b34ef8ae3651a519b8ff8c (127.3.221.67:34091)
I20260811 11:54:33.949653 3145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e6a9acdab84bbd9d5e83b3ee8a51ec" candidate_uuid: "5e5cdb3dba704dc88060df99509839a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c536ac7380ea4a75bccf2363c79014e0"
I20260811 11:54:33.949704 3359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e6a9acdab84bbd9d5e83b3ee8a51ec" candidate_uuid: "5e5cdb3dba704dc88060df99509839a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6becc20b11b34ef8ae3651a519b8ff8c"
I20260811 11:54:33.949784 3145 raft_consensus.cc:3037] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P c536ac7380ea4a75bccf2363c79014e0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:33.949815 3359 raft_consensus.cc:3037] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 6becc20b11b34ef8ae3651a519b8ff8c [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:33.951056 3359 raft_consensus.cc:2454] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 6becc20b11b34ef8ae3651a519b8ff8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5cdb3dba704dc88060df99509839a1 in term 1.
I20260811 11:54:33.951274 3145 raft_consensus.cc:2454] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P c536ac7380ea4a75bccf2363c79014e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5cdb3dba704dc88060df99509839a1 in term 1.
I20260811 11:54:33.951381 2838 leader_election.cc:302] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5cdb3dba704dc88060df99509839a1, 6becc20b11b34ef8ae3651a519b8ff8c; no voters:
I20260811 11:54:33.951578 3471 raft_consensus.cc:2781] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:33.951843 3471 raft_consensus.cc:686] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1 [term 1 LEADER]: Becoming Leader. State: Replica: 5e5cdb3dba704dc88060df99509839a1, State: Running, Role: LEADER
I20260811 11:54:33.952211 3471 consensus_queue.cc:237] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c536ac7380ea4a75bccf2363c79014e0" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 35343 } } peers { permanent_uuid: "5e5cdb3dba704dc88060df99509839a1" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 36713 } } peers { permanent_uuid: "6becc20b11b34ef8ae3651a519b8ff8c" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 34091 } }
I20260811 11:54:33.955694 2716 catalog_manager.cc:5495] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e5cdb3dba704dc88060df99509839a1 (127.3.221.65). New cstate: current_term: 1 leader_uuid: "5e5cdb3dba704dc88060df99509839a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c536ac7380ea4a75bccf2363c79014e0" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 35343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e5cdb3dba704dc88060df99509839a1" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 36713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6becc20b11b34ef8ae3651a519b8ff8c" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 34091 } health_report { overall_health: UNKNOWN } } }
I20260811 11:54:33.979199 3460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:33.975632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60266 (local address 127.3.221.65:36713)
0811 11:54:33.975772 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:33.975777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:33.975790 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:54:33.975884 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:33.975887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:33.975942 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:33.976039 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:54:33.976275 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.976799 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.977733 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:33.977953 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:33.978042 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:33.978336 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:54:33.978792 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:54:33.978912 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:33.978993 (+ 81us) server_negotiation.cc:299] Negotiation successful
0811 11:54:33.979055 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 11:54:33.985575 3359 raft_consensus.cc:1261] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 6becc20b11b34ef8ae3651a519b8ff8c [term 1 FOLLOWER]: Refusing update from remote peer 5e5cdb3dba704dc88060df99509839a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:54:33.985575 3145 raft_consensus.cc:1261] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P c536ac7380ea4a75bccf2363c79014e0 [term 1 FOLLOWER]: Refusing update from remote peer 5e5cdb3dba704dc88060df99509839a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:54:33.986327 3471 consensus_queue.cc:1035] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6becc20b11b34ef8ae3651a519b8ff8c" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:54:33.986610 3483 consensus_queue.cc:1035] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c536ac7380ea4a75bccf2363c79014e0" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 35343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:54:33.997349 3493 mvcc.cc:205] Tried to move back new op lower bound from 7317296226236403712 to 7317296226111242240. Current Snapshot: MvccSnapshot[applied={T|T < 7317296226236403712}]
I20260811 11:54:34.005357 2716 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:54:34.005577 2716 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:54:34.008276 2716 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f43ca40d79b480bb381014c35ae9946: Sending DeleteTablet for 3 replicas of tablet 05e6a9acdab84bbd9d5e83b3ee8a51ec
I20260811 11:54:34.009188 2910 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e6a9acdab84bbd9d5e83b3ee8a51ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:54:34 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:60256
I20260811 11:54:34.009245 3116 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e6a9acdab84bbd9d5e83b3ee8a51ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:54:34 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:56140
I20260811 11:54:34.009285 3329 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e6a9acdab84bbd9d5e83b3ee8a51ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:54:34 UTC) from {username='kudu', principal='kudu/127.3.221.126@KRBTEST.COM'} at 127.0.0.1:39210
I20260811 11:54:34.009996 3504 tablet_replica.cc:330] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1: stopping tablet replica
I20260811 11:54:34.010012 3502 tablet_replica.cc:330] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 6becc20b11b34ef8ae3651a519b8ff8c: stopping tablet replica
I20260811 11:54:34.010373 3504 raft_consensus.cc:2229] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 5e5cdb3dba704dc88060df99509839a1 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:54:34.010380 3502 raft_consensus.cc:2229] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 6becc20b11b34ef8ae3651a519b8ff8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:54:34.010599 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 2811
I20260811 11:54:34.010730 3502 raft_consensus.cc:2258] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 6becc20b11b34ef8ae3651a519b8ff8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:54:34.016865 3502 ts_tablet_manager.cc:1905] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 6becc20b11b34ef8ae3651a519b8ff8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:54:34.020879 3507 tablet_replica.cc:330] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P c536ac7380ea4a75bccf2363c79014e0: stopping tablet replica
I20260811 11:54:34.021320 3507 raft_consensus.cc:2229] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P c536ac7380ea4a75bccf2363c79014e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:54:34.021734 3507 raft_consensus.cc:2258] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P c536ac7380ea4a75bccf2363c79014e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:54:34.022799 3502 ts_tablet_manager.cc:1918] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 6becc20b11b34ef8ae3651a519b8ff8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:54:34.022979 3502 log.cc:1196] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 6becc20b11b34ef8ae3651a519b8ff8c: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/05e6a9acdab84bbd9d5e83b3ee8a51ec
I20260811 11:54:34.023404 3502 ts_tablet_manager.cc:1939] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P 6becc20b11b34ef8ae3651a519b8ff8c: Deleting consensus metadata
I20260811 11:54:34.023947 3507 ts_tablet_manager.cc:1905] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P c536ac7380ea4a75bccf2363c79014e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:54:34.024927 2699 catalog_manager.cc:4841] TS 6becc20b11b34ef8ae3651a519b8ff8c (127.3.221.67:34091): tablet 05e6a9acdab84bbd9d5e83b3ee8a51ec (table test-table [id=64ab0895857f4ff987d12b5176e7c72c]) successfully deleted
I20260811 11:54:34.026770 3507 ts_tablet_manager.cc:1918] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P c536ac7380ea4a75bccf2363c79014e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:54:34.026906 3507 log.cc:1196] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P c536ac7380ea4a75bccf2363c79014e0: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/05e6a9acdab84bbd9d5e83b3ee8a51ec
I20260811 11:54:34.027205 3507 ts_tablet_manager.cc:1939] T 05e6a9acdab84bbd9d5e83b3ee8a51ec P c536ac7380ea4a75bccf2363c79014e0: Deleting consensus metadata
I20260811 11:54:34.028592 2699 catalog_manager.cc:4841] TS c536ac7380ea4a75bccf2363c79014e0 (127.3.221.66:35343): tablet 05e6a9acdab84bbd9d5e83b3ee8a51ec (table test-table [id=64ab0895857f4ff987d12b5176e7c72c]) successfully deleted
W20260811 11:54:34.029531 2698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.221.65:36713 (error 108)
I20260811 11:54:34.030346 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 3016
I20260811 11:54:34.030767 3458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:34.030504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:36713 (local address 127.0.0.1:60268)
0811 11:54:34.030619 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:34.030688 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.65:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260811 11:54:34.033396 2698 catalog_manager.cc:4584] TS 5e5cdb3dba704dc88060df99509839a1 (127.3.221.65:36713): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 05e6a9acdab84bbd9d5e83b3ee8a51ec: Network error: Client connection negotiation failed: client connection to 127.3.221.65:36713: connect: Connection refused (error 111)
I20260811 11:54:34.042187 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 3228
I20260811 11:54:34.070101 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 2658
2026-08-11T11:54:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2623 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-11T11:54:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:54:34Z Disabled control of system clock
I20260811 11:54:34.116163 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:36457
--webserver_interface=127.3.221.126
--webserver_port=0
--builtin_ntp_servers=127.3.221.84:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:36457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 11:54:34.240057 3515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:34.240399 3515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:34.240454 3515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:34.244620 3515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:54:34.244699 3515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:34.244721 3515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:34.244745 3515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:54:34.244763 3515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:54:34.250293 3515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:36457
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:36457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:34.251595 3515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:34.253082 3515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:34.261858 3528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:34.261826 3533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:34.261916 3515 server_base.cc:1034] running on GCE node
W20260811 11:54:34.261826 3530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:34.262725 3515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:34.263911 3515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:34.265103 3515 hybrid_clock.cc:648] HybridClock initialized: now 1786449274265079 us; error 36 us; skew 500 ppm
I20260811 11:54:34.270457 3515 webserver.cc:460] Webserver started at http://127.3.221.126:46527/ using document root <none> and password file <none>
I20260811 11:54:34.271133 3515 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:34.271234 3515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:34.271481 3515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:34.273769 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "63007ea0d8524b908c15ce330b071f0e"
format_stamp: "Formatted at 2026-08-11 11:54:34 on dist-test-slave-bndk"
server_key: "58459f3d24f13059492e81747dafcbbd"
server_key_iv: "4146bf911ee20b91505a9c76bd42e53f"
server_key_version: "encryptionkey@0"
I20260811 11:54:34.274371 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "63007ea0d8524b908c15ce330b071f0e"
format_stamp: "Formatted at 2026-08-11 11:54:34 on dist-test-slave-bndk"
server_key: "58459f3d24f13059492e81747dafcbbd"
server_key_iv: "4146bf911ee20b91505a9c76bd42e53f"
server_key_version: "encryptionkey@0"
I20260811 11:54:34.279078 3515 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 11:54:34.281997 3540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:34.283567 3515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:54:34.283712 3515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "63007ea0d8524b908c15ce330b071f0e"
format_stamp: "Formatted at 2026-08-11 11:54:34 on dist-test-slave-bndk"
server_key: "58459f3d24f13059492e81747dafcbbd"
server_key_iv: "4146bf911ee20b91505a9c76bd42e53f"
server_key_version: "encryptionkey@0"
I20260811 11:54:34.283835 3515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:34.304194 3515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:34.305075 3515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:34.305332 3515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:34.314776 3515 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:36457
I20260811 11:54:34.314805 3627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:36457 every 8 connection(s)
I20260811 11:54:34.315918 3515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:54:34.319449 3630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:34.324591 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 3515
I20260811 11:54:34.324731 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:54:34.325038 3957 external_mini_cluster.cc:1428] Setting key 726fb5170edb1a736304ab5e5785e197
I20260811 11:54:34.326646 3630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e: Bootstrap starting.
I20260811 11:54:34.329571 3630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:34.330389 3630 log.cc:824] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:34.332429 3630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e: No bootstrap required, opened a new log
I20260811 11:54:34.335266 3638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:34.326924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42506 (local address 127.3.221.126:36457)
0811 11:54:34.327750 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:34.327774 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:34.327818 (+ 44us) server_negotiation.cc:407] Connection header received
0811 11:54:34.328597 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:34.328618 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:34.329058 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:34.329457 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:34.329924 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:34.330978 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:34.331910 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:34.332466 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:34.332622 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:34.333501 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:34.333506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:34.333521 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:34.333928 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:34.333987 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:34.334008 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:34.334100 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:34.334235 (+ 135us) server_negotiation.cc:299] Negotiation successful
0811 11:54:34.334519 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":150,"threads_started":1}
I20260811 11:54:34.336350 3630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63007ea0d8524b908c15ce330b071f0e" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36457 } }
I20260811 11:54:34.336591 3630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:34.336647 3630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63007ea0d8524b908c15ce330b071f0e, State: Initialized, Role: FOLLOWER
I20260811 11:54:34.337095 3630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63007ea0d8524b908c15ce330b071f0e" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36457 } }
I20260811 11:54:34.337275 3630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:54:34.337349 3630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:54:34.337495 3630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:34.338541 3630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63007ea0d8524b908c15ce330b071f0e" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36457 } }
I20260811 11:54:34.338899 3630 leader_election.cc:302] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63007ea0d8524b908c15ce330b071f0e; no voters:
I20260811 11:54:34.339282 3630 leader_election.cc:288] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:54:34.339378 3640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:34.339555 3640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e [term 1 LEADER]: Becoming Leader. State: Replica: 63007ea0d8524b908c15ce330b071f0e, State: Running, Role: LEADER
I20260811 11:54:34.339958 3640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63007ea0d8524b908c15ce330b071f0e" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36457 } }
I20260811 11:54:34.340699 3630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:54:34.341907 3644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63007ea0d8524b908c15ce330b071f0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63007ea0d8524b908c15ce330b071f0e" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36457 } } }
I20260811 11:54:34.342052 3644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:34.342370 3643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63007ea0d8524b908c15ce330b071f0e. Latest consensus state: current_term: 1 leader_uuid: "63007ea0d8524b908c15ce330b071f0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63007ea0d8524b908c15ce330b071f0e" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 36457 } } }
I20260811 11:54:34.342564 3643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:34.342453 3654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:54:34.345921 3654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:54:34.352366 3654 catalog_manager.cc:1347] Generated new cluster ID: b8e170c4851a44b0bdbf7bc2115059e4
I20260811 11:54:34.352468 3654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:54:34.368217 3654 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:54:34.369244 3654 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:54:34.390213 3654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e: Generated new TSK 0
I20260811 11:54:34.391181 3654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:54:34.402439 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.65:0
--local_ip_for_outbound_sockets=127.3.221.65
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36457
--builtin_ntp_servers=127.3.221.84:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 11:54:34.525303 3670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:34.525569 3670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:34.525607 3670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:34.529577 3670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:34.529628 3670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:34.529685 3670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.65
I20260811 11:54:34.534480 3670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.221.65
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.65
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:34.535574 3670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:34.536954 3670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:34.547211 3677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:34.547413 3670 server_base.cc:1034] running on GCE node
W20260811 11:54:34.547214 3676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:34.547618 3680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:34.548065 3670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:34.548753 3670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:34.549947 3670 hybrid_clock.cc:648] HybridClock initialized: now 1786449274549923 us; error 38 us; skew 500 ppm
I20260811 11:54:34.551999 3670 webserver.cc:460] Webserver started at http://127.3.221.65:46569/ using document root <none> and password file <none>
I20260811 11:54:34.552693 3670 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:34.552781 3670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:34.553021 3670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:34.555271 3670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-0/data/instance:
uuid: "739234e9520c416f92b38176137e07df"
format_stamp: "Formatted at 2026-08-11 11:54:34 on dist-test-slave-bndk"
server_key: "bdf43b6a110b0d4b3c01b9241d6b64ce"
server_key_iv: "127f910d7aba92a6b4eea12a98b9efd3"
server_key_version: "encryptionkey@0"
I20260811 11:54:34.555867 3670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance:
uuid: "739234e9520c416f92b38176137e07df"
format_stamp: "Formatted at 2026-08-11 11:54:34 on dist-test-slave-bndk"
server_key: "bdf43b6a110b0d4b3c01b9241d6b64ce"
server_key_iv: "127f910d7aba92a6b4eea12a98b9efd3"
server_key_version: "encryptionkey@0"
I20260811 11:54:34.560055 3670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260811 11:54:34.563282 3690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:34.564718 3670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 11:54:34.564852 3670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-0/wal
uuid: "739234e9520c416f92b38176137e07df"
format_stamp: "Formatted at 2026-08-11 11:54:34 on dist-test-slave-bndk"
server_key: "bdf43b6a110b0d4b3c01b9241d6b64ce"
server_key_iv: "127f910d7aba92a6b4eea12a98b9efd3"
server_key_version: "encryptionkey@0"
I20260811 11:54:34.564972 3670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:34.588241 3670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:34.589010 3670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:34.589200 3670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:34.589874 3670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:54:34.590934 3670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:54:34.590986 3670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:34.591043 3670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:54:34.591073 3670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:34.600965 3670 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.65:45309
I20260811 11:54:34.601037 3879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.65:45309 every 8 connection(s)
I20260811 11:54:34.602002 3670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-0/data/info.pb
I20260811 11:54:34.611047 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 3670
I20260811 11:54:34.611183 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-0/wal/instance
I20260811 11:54:34.611488 3957 external_mini_cluster.cc:1428] Setting key 97de11403b212761162b930e37414ee4
I20260811 11:54:34.612011 3638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:34.603944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.65:53555 (local address 127.3.221.126:36457)
0811 11:54:34.604080 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:34.604083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:34.605164 (+ 1081us) server_negotiation.cc:407] Connection header received
0811 11:54:34.606728 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:34.606733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:34.606821 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:34.606958 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:34.607918 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:34.608495 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:34.609806 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:34.610045 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:34.610137 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:34.611459 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:34.611463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:34.611466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:34.611485 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:34.611510 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:34.611522 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:34.611607 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:34.611810 (+ 203us) server_negotiation.cc:299] Negotiation successful
0811 11:54:34.611881 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 11:54:34.613060 3883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:34.604329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36457 (local address 127.3.221.65:53555)
0811 11:54:34.605013 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:34.605057 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:34.606392 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:34.607130 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:34.607137 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:34.607227 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:54:34.607749 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:34.607762 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:34.608648 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:34.608651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:34.609664 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:34.609677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:34.610224 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:34.610227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:34.610390 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:34.611102 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:54:34.611104 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:54:34.611151 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:54:34.611258 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:54:34.611646 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:54:34.611657 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:54:34.611663 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 11:54:34.611788 (+ 125us) client_negotiation.cc:240] Negotiation successful
0811 11:54:34.612137 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":112,"threads_started":1}
I20260811 11:54:34.613641 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.66:0
--local_ip_for_outbound_sockets=127.3.221.66
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36457
--builtin_ntp_servers=127.3.221.84:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 11:54:34.614284 3881 heartbeater.cc:347] Connected to a master server at 127.3.221.126:36457
I20260811 11:54:34.614647 3881 heartbeater.cc:464] Registering TS with master...
I20260811 11:54:34.615216 3881 heartbeater.cc:511] Master 127.3.221.126:36457 requested a full tablet report, sending...
I20260811 11:54:34.616792 3560 ts_manager.cc:194] Registered new tserver with Master: 739234e9520c416f92b38176137e07df (127.3.221.65:45309)
I20260811 11:54:34.618017 3560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.221.65:53555
W20260811 11:54:34.737387 3886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:34.737736 3886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:34.737811 3886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:34.744252 3886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:34.744378 3886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:34.744465 3886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.66
I20260811 11:54:34.751627 3886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.221.66
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.66
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:34.753401 3886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:34.755257 3886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:34.768728 3893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:34.768728 3892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:34.769181 3897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:34.769182 3886 server_base.cc:1034] running on GCE node
I20260811 11:54:34.769883 3886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:34.770853 3886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:34.772156 3886 hybrid_clock.cc:648] HybridClock initialized: now 1786449274772069 us; error 104 us; skew 500 ppm
I20260811 11:54:34.775485 3886 webserver.cc:460] Webserver started at http://127.3.221.66:38715/ using document root <none> and password file <none>
I20260811 11:54:34.776553 3886 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:34.776757 3886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:34.777070 3886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:34.780325 3886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-1/data/instance:
uuid: "e688acdd9320482aa68b5a799632d805"
format_stamp: "Formatted at 2026-08-11 11:54:34 on dist-test-slave-bndk"
server_key: "9f7f7e64bf165d22e6cd1c12ebd9d748"
server_key_iv: "5bcf75b0ed65e02f1a59b432abece3ab"
server_key_version: "encryptionkey@0"
I20260811 11:54:34.781153 3886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance:
uuid: "e688acdd9320482aa68b5a799632d805"
format_stamp: "Formatted at 2026-08-11 11:54:34 on dist-test-slave-bndk"
server_key: "9f7f7e64bf165d22e6cd1c12ebd9d748"
server_key_iv: "5bcf75b0ed65e02f1a59b432abece3ab"
server_key_version: "encryptionkey@0"
I20260811 11:54:34.786918 3886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260811 11:54:34.790689 3910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:34.792132 3886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 11:54:34.792260 3886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-1/wal
uuid: "e688acdd9320482aa68b5a799632d805"
format_stamp: "Formatted at 2026-08-11 11:54:34 on dist-test-slave-bndk"
server_key: "9f7f7e64bf165d22e6cd1c12ebd9d748"
server_key_iv: "5bcf75b0ed65e02f1a59b432abece3ab"
server_key_version: "encryptionkey@0"
I20260811 11:54:34.792397 3886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:34.821301 3886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:34.822085 3886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:34.822273 3886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:34.822932 3886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:54:34.824105 3886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:54:34.824160 3886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:34.824203 3886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:54:34.824218 3886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:34.833712 3886 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.66:44821
I20260811 11:54:34.833743 4116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.66:44821 every 8 connection(s)
I20260811 11:54:34.834693 3886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-1/data/info.pb
I20260811 11:54:34.843825 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 3886
I20260811 11:54:34.843966 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-1/wal/instance
I20260811 11:54:34.844244 3957 external_mini_cluster.cc:1428] Setting key b555544e953c7708cce73638c1f3fd62
I20260811 11:54:34.844769 3638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:34.836702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:55385 (local address 127.3.221.126:36457)
0811 11:54:34.836854 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:34.836858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:34.838054 (+ 1196us) server_negotiation.cc:407] Connection header received
0811 11:54:34.839590 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:34.839594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:34.839661 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:34.839742 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:34.840754 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:34.841432 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:34.842715 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:34.842989 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:34.843083 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:34.844267 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:34.844271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:34.844274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:34.844297 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:34.844319 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:34.844329 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:34.844397 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:34.844546 (+ 149us) server_negotiation.cc:299] Negotiation successful
0811 11:54:34.844614 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 11:54:34.845953 4120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:34.837082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36457 (local address 127.3.221.66:55385)
0811 11:54:34.837878 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:34.837923 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:34.839341 (+ 1418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:34.839956 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:34.839964 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:34.840061 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:54:34.840600 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:34.840614 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:34.841566 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:34.841568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:34.842574 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:34.842587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:34.843144 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:34.843148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:34.843314 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:34.843958 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:54:34.843960 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:54:34.844022 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:54:34.844110 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:54:34.844399 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:54:34.844410 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:54:34.844417 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 11:54:34.844542 (+ 125us) client_negotiation.cc:240] Negotiation successful
0811 11:54:34.844880 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":179,"threads_started":1}
I20260811 11:54:34.846997 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.221.67:0
--local_ip_for_outbound_sockets=127.3.221.67
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36457
--builtin_ntp_servers=127.3.221.84:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 11:54:34.847291 4117 heartbeater.cc:347] Connected to a master server at 127.3.221.126:36457
I20260811 11:54:34.847559 4117 heartbeater.cc:464] Registering TS with master...
I20260811 11:54:34.848222 4117 heartbeater.cc:511] Master 127.3.221.126:36457 requested a full tablet report, sending...
I20260811 11:54:34.849421 3560 ts_manager.cc:194] Registered new tserver with Master: e688acdd9320482aa68b5a799632d805 (127.3.221.66:44821)
I20260811 11:54:34.850013 3560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.221.66:55385
W20260811 11:54:34.970476 4124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:34.970781 4124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:34.970847 4124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:34.975028 4124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:34.975097 4124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:34.975176 4124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.221.67
I20260811 11:54:34.980178 4124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.221.67
--webserver_port=0
--tserver_master_addrs=127.3.221.126:36457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.221.67
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:34.981426 4124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:34.982834 4124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:34.992813 4131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:34.992813 4130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:34.993188 4124 server_base.cc:1034] running on GCE node
W20260811 11:54:34.993476 4133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:34.993930 4124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:34.994643 4124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:34.997923 4124 hybrid_clock.cc:648] HybridClock initialized: now 1786449274997888 us; error 62 us; skew 500 ppm
I20260811 11:54:35.000051 4124 webserver.cc:460] Webserver started at http://127.3.221.67:35143/ using document root <none> and password file <none>
I20260811 11:54:35.000674 4124 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:35.000749 4124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:35.000967 4124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:35.003218 4124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-2/data/instance:
uuid: "e7f5f7d65b4646eba609d8b4ddca23fa"
format_stamp: "Formatted at 2026-08-11 11:54:34 on dist-test-slave-bndk"
server_key: "1da7730daed77d820bfa5c8ef459e1b6"
server_key_iv: "a09106be3e27092bdc7c8eb0b6600171"
server_key_version: "encryptionkey@0"
I20260811 11:54:35.003810 4124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance:
uuid: "e7f5f7d65b4646eba609d8b4ddca23fa"
format_stamp: "Formatted at 2026-08-11 11:54:34 on dist-test-slave-bndk"
server_key: "1da7730daed77d820bfa5c8ef459e1b6"
server_key_iv: "a09106be3e27092bdc7c8eb0b6600171"
server_key_version: "encryptionkey@0"
I20260811 11:54:35.007747 4124 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 11:54:35.010390 4142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:35.011739 4124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:54:35.011860 4124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-2/wal
uuid: "e7f5f7d65b4646eba609d8b4ddca23fa"
format_stamp: "Formatted at 2026-08-11 11:54:34 on dist-test-slave-bndk"
server_key: "1da7730daed77d820bfa5c8ef459e1b6"
server_key_iv: "a09106be3e27092bdc7c8eb0b6600171"
server_key_version: "encryptionkey@0"
I20260811 11:54:35.011977 4124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:35.025338 4124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:35.026201 4124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:35.026415 4124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:35.027057 4124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 11:54:35.028228 4124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 11:54:35.028299 4124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:35.028374 4124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 11:54:35.028400 4124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:35.037756 4124 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.67:35197
I20260811 11:54:35.037775 4335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.67:35197 every 8 connection(s)
I20260811 11:54:35.038748 4124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-2/data/info.pb
I20260811 11:54:35.046027 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 4124
I20260811 11:54:35.046170 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-2/wal/instance
I20260811 11:54:35.046450 3957 external_mini_cluster.cc:1428] Setting key 378d592784fd57a821d076a4de73cb9c
I20260811 11:54:35.048774 3638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:35.040769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.67:41097 (local address 127.3.221.126:36457)
0811 11:54:35.040925 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:35.040929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:35.042147 (+ 1218us) server_negotiation.cc:407] Connection header received
0811 11:54:35.043655 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:35.043658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:35.043721 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:35.043801 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:35.044758 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.045339 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.046603 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.046884 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.046998 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:35.048213 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:35.048218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:35.048221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:35.048234 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:35.048258 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:35.048267 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:35.048362 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:35.048530 (+ 168us) server_negotiation.cc:299] Negotiation successful
0811 11:54:35.048596 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 11:54:35.049824 4339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:35.041177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:36457 (local address 127.3.221.67:41097)
0811 11:54:35.041981 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:35.042026 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:35.043424 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:35.044002 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:35.044009 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:35.044098 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:54:35.044608 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:35.044621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.045494 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.045498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:35.046475 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:35.046487 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.047055 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.047058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:35.047214 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:35.047904 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 11:54:35.047906 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:54:35.047966 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 11:54:35.048045 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:54:35.048368 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:54:35.048375 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:54:35.048381 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 11:54:35.048528 (+ 147us) client_negotiation.cc:240] Negotiation successful
0811 11:54:35.048943 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":124,"threads_started":1}
I20260811 11:54:35.050944 4336 heartbeater.cc:347] Connected to a master server at 127.3.221.126:36457
I20260811 11:54:35.051180 4336 heartbeater.cc:464] Registering TS with master...
I20260811 11:54:35.051752 4336 heartbeater.cc:511] Master 127.3.221.126:36457 requested a full tablet report, sending...
I20260811 11:54:35.052827 3560 ts_manager.cc:194] Registered new tserver with Master: e7f5f7d65b4646eba609d8b4ddca23fa (127.3.221.67:35197)
I20260811 11:54:35.053439 3560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.221.67:41097
I20260811 11:54:35.060883 3957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 11:54:35.068027 3638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:35.063006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42510 (local address 127.3.221.126:36457)
0811 11:54:35.063313 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:35.063320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:35.063558 (+ 238us) server_negotiation.cc:407] Connection header received
0811 11:54:35.063761 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:35.063764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:35.063822 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:35.063894 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:35.064394 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.065000 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.066337 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.066598 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.066706 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:35.067554 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:35.067558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:35.067560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:35.067574 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:35.067592 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:35.067600 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:35.067665 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:35.067758 (+ 93us) server_negotiation.cc:299] Negotiation successful
0811 11:54:35.067834 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260811 11:54:35.072453 3560 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 11:54:35.076510 3560 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 11:54:35.091898 4353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:35.085981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:35197 (local address 127.0.0.1:50704)
0811 11:54:35.086825 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:35.086878 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:35.087027 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:35.087882 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:35.087889 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:35.087912 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:35.088095 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:35.088105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.089267 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.089270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:35.090340 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:35.090349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.091293 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.091296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:35.091414 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:35.091456 (+ 42us) client_negotiation.cc:769] Sending connection context
0811 11:54:35.091571 (+ 115us) client_negotiation.cc:240] Negotiation successful
0811 11:54:35.091706 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":1,"spinlock_wait_cycles":3200,"thread_start_us":84,"threads_started":1}
I20260811 11:54:35.091910 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:35.086141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50704 (local address 127.3.221.67:35197)
0811 11:54:35.087317 (+ 1176us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:35.087329 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:35.087371 (+ 42us) server_negotiation.cc:407] Connection header received
0811 11:54:35.087457 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:35.087462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:35.087700 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:35.087865 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:35.088232 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.089141 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.090483 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.091093 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.091194 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:35.091424 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:35.091615 (+ 191us) server_negotiation.cc:299] Negotiation successful
0811 11:54:35.091721 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1074,"thread_start_us":181,"threads_started":1}
I20260811 11:54:35.092727 4354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:35.086371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:45309 (local address 127.0.0.1:45614)
0811 11:54:35.087243 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:35.087255 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:35.087356 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:35.087918 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:35.087922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:35.087958 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:35.088115 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:35.088123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.089904 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.089907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:35.091068 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:35.091078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.092329 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.092334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:35.092434 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:35.092463 (+ 29us) client_negotiation.cc:769] Sending connection context
0811 11:54:35.092520 (+ 57us) client_negotiation.cc:240] Negotiation successful
0811 11:54:35.092590 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":44,"spinlock_wait_cycles":8832,"thread_start_us":99,"threads_started":1}
I20260811 11:54:35.092860 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:35.086134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.66:44821 (local address 127.0.0.1:38346)
0811 11:54:35.086877 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:35.086888 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:35.087027 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:35.088164 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:35.088167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:35.088182 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:35.088268 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:35.088274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.090502 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.090506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:35.091570 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:35.091579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.092514 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.092518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:35.092598 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:35.092617 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 11:54:35.092660 (+ 43us) client_negotiation.cc:240] Negotiation successful
0811 11:54:35.092712 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"spinlock_wait_cycles":3200,"thread_start_us":121,"threads_started":1}
I20260811 11:54:35.092815 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:35.086811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45614 (local address 127.3.221.65:45309)
0811 11:54:35.087188 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:35.087195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:35.087378 (+ 183us) server_negotiation.cc:407] Connection header received
0811 11:54:35.087478 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:35.087483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:35.087695 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:35.088080 (+ 385us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:35.088223 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.089720 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.091208 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.091988 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.092322 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:35.092512 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:35.092566 (+ 54us) server_negotiation.cc:299] Negotiation successful
0811 11:54:35.092661 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":98,"threads_started":1}
I20260811 11:54:35.094231 4361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:35.086771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38346 (local address 127.3.221.66:44821)
0811 11:54:35.087679 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:35.087721 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:35.087745 (+ 24us) server_negotiation.cc:407] Connection header received
0811 11:54:35.087829 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:35.087834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:35.088012 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:35.088537 (+ 525us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:35.088957 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.090360 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.091701 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.092382 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.093000 (+ 618us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:35.093903 (+ 903us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:35.093958 (+ 55us) server_negotiation.cc:299] Negotiation successful
0811 11:54:35.094066 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":118,"threads_started":1}
I20260811 11:54:35.095530 4229 tablet_service.cc:1467] Processing CreateTablet for tablet a9c308a73dcb4ca49fa6036663c0009e (DEFAULT_TABLE table=test-table [id=e1cffe860bee447bbf7fb8a819ac3ac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:54:35.095603 3773 tablet_service.cc:1467] Processing CreateTablet for tablet a9c308a73dcb4ca49fa6036663c0009e (DEFAULT_TABLE table=test-table [id=e1cffe860bee447bbf7fb8a819ac3ac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:54:35.096632 3773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9c308a73dcb4ca49fa6036663c0009e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:35.096650 4229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9c308a73dcb4ca49fa6036663c0009e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:35.096974 3996 tablet_service.cc:1467] Processing CreateTablet for tablet a9c308a73dcb4ca49fa6036663c0009e (DEFAULT_TABLE table=test-table [id=e1cffe860bee447bbf7fb8a819ac3ac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 11:54:35.098232 3996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9c308a73dcb4ca49fa6036663c0009e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:35.103749 4363 tablet_bootstrap.cc:492] T a9c308a73dcb4ca49fa6036663c0009e P 739234e9520c416f92b38176137e07df: Bootstrap starting.
I20260811 11:54:35.104718 4365 tablet_bootstrap.cc:492] T a9c308a73dcb4ca49fa6036663c0009e P e7f5f7d65b4646eba609d8b4ddca23fa: Bootstrap starting.
I20260811 11:54:35.105576 4366 tablet_bootstrap.cc:492] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805: Bootstrap starting.
I20260811 11:54:35.106094 4363 tablet_bootstrap.cc:654] T a9c308a73dcb4ca49fa6036663c0009e P 739234e9520c416f92b38176137e07df: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:35.106707 4365 tablet_bootstrap.cc:654] T a9c308a73dcb4ca49fa6036663c0009e P e7f5f7d65b4646eba609d8b4ddca23fa: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:35.107041 4363 log.cc:824] T a9c308a73dcb4ca49fa6036663c0009e P 739234e9520c416f92b38176137e07df: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:35.107390 4365 log.cc:824] T a9c308a73dcb4ca49fa6036663c0009e P e7f5f7d65b4646eba609d8b4ddca23fa: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:35.107771 4366 tablet_bootstrap.cc:654] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:35.108616 4366 log.cc:824] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:35.109508 4365 tablet_bootstrap.cc:492] T a9c308a73dcb4ca49fa6036663c0009e P e7f5f7d65b4646eba609d8b4ddca23fa: No bootstrap required, opened a new log
I20260811 11:54:35.109529 4363 tablet_bootstrap.cc:492] T a9c308a73dcb4ca49fa6036663c0009e P 739234e9520c416f92b38176137e07df: No bootstrap required, opened a new log
I20260811 11:54:35.109704 4365 ts_tablet_manager.cc:1397] T a9c308a73dcb4ca49fa6036663c0009e P e7f5f7d65b4646eba609d8b4ddca23fa: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260811 11:54:35.109709 4363 ts_tablet_manager.cc:1397] T a9c308a73dcb4ca49fa6036663c0009e P 739234e9520c416f92b38176137e07df: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 11:54:35.111717 4366 tablet_bootstrap.cc:492] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805: No bootstrap required, opened a new log
I20260811 11:54:35.111979 4366 ts_tablet_manager.cc:1397] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260811 11:54:35.113186 4365 raft_consensus.cc:348] T a9c308a73dcb4ca49fa6036663c0009e P e7f5f7d65b4646eba609d8b4ddca23fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f5f7d65b4646eba609d8b4ddca23fa" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 35197 } } peers { permanent_uuid: "e688acdd9320482aa68b5a799632d805" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44821 } } peers { permanent_uuid: "739234e9520c416f92b38176137e07df" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45309 } }
I20260811 11:54:35.113483 4365 raft_consensus.cc:374] T a9c308a73dcb4ca49fa6036663c0009e P e7f5f7d65b4646eba609d8b4ddca23fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:35.113540 4365 raft_consensus.cc:729] T a9c308a73dcb4ca49fa6036663c0009e P e7f5f7d65b4646eba609d8b4ddca23fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f5f7d65b4646eba609d8b4ddca23fa, State: Initialized, Role: FOLLOWER
I20260811 11:54:35.114068 4365 consensus_queue.cc:260] T a9c308a73dcb4ca49fa6036663c0009e P e7f5f7d65b4646eba609d8b4ddca23fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f5f7d65b4646eba609d8b4ddca23fa" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 35197 } } peers { permanent_uuid: "e688acdd9320482aa68b5a799632d805" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44821 } } peers { permanent_uuid: "739234e9520c416f92b38176137e07df" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45309 } }
I20260811 11:54:35.114773 4336 heartbeater.cc:503] Master 127.3.221.126:36457 was elected leader, sending a full tablet report...
I20260811 11:54:35.114694 4363 raft_consensus.cc:348] T a9c308a73dcb4ca49fa6036663c0009e P 739234e9520c416f92b38176137e07df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f5f7d65b4646eba609d8b4ddca23fa" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 35197 } } peers { permanent_uuid: "e688acdd9320482aa68b5a799632d805" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44821 } } peers { permanent_uuid: "739234e9520c416f92b38176137e07df" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45309 } }
I20260811 11:54:35.114948 4363 raft_consensus.cc:374] T a9c308a73dcb4ca49fa6036663c0009e P 739234e9520c416f92b38176137e07df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:35.115003 4363 raft_consensus.cc:729] T a9c308a73dcb4ca49fa6036663c0009e P 739234e9520c416f92b38176137e07df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 739234e9520c416f92b38176137e07df, State: Initialized, Role: FOLLOWER
I20260811 11:54:35.115175 4365 ts_tablet_manager.cc:1428] T a9c308a73dcb4ca49fa6036663c0009e P e7f5f7d65b4646eba609d8b4ddca23fa: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260811 11:54:35.115660 4363 consensus_queue.cc:260] T a9c308a73dcb4ca49fa6036663c0009e P 739234e9520c416f92b38176137e07df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f5f7d65b4646eba609d8b4ddca23fa" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 35197 } } peers { permanent_uuid: "e688acdd9320482aa68b5a799632d805" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44821 } } peers { permanent_uuid: "739234e9520c416f92b38176137e07df" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45309 } }
I20260811 11:54:35.115877 4366 raft_consensus.cc:348] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f5f7d65b4646eba609d8b4ddca23fa" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 35197 } } peers { permanent_uuid: "e688acdd9320482aa68b5a799632d805" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44821 } } peers { permanent_uuid: "739234e9520c416f92b38176137e07df" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45309 } }
I20260811 11:54:35.116148 4366 raft_consensus.cc:374] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:35.116196 4366 raft_consensus.cc:729] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e688acdd9320482aa68b5a799632d805, State: Initialized, Role: FOLLOWER
I20260811 11:54:35.116600 4366 consensus_queue.cc:260] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f5f7d65b4646eba609d8b4ddca23fa" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 35197 } } peers { permanent_uuid: "e688acdd9320482aa68b5a799632d805" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44821 } } peers { permanent_uuid: "739234e9520c416f92b38176137e07df" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45309 } }
I20260811 11:54:35.116808 4363 ts_tablet_manager.cc:1428] T a9c308a73dcb4ca49fa6036663c0009e P 739234e9520c416f92b38176137e07df: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260811 11:54:35.117321 3881 heartbeater.cc:503] Master 127.3.221.126:36457 was elected leader, sending a full tablet report...
I20260811 11:54:35.117599 4366 ts_tablet_manager.cc:1428] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260811 11:54:35.117851 4117 heartbeater.cc:503] Master 127.3.221.126:36457 was elected leader, sending a full tablet report...
W20260811 11:54:35.289795 4337 tablet.cc:2369] T a9c308a73dcb4ca49fa6036663c0009e P e7f5f7d65b4646eba609d8b4ddca23fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 11:54:35.335909 4118 tablet.cc:2369] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:54:35.338374 4379 raft_consensus.cc:482] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 11:54:35.338616 4379 raft_consensus.cc:504] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f5f7d65b4646eba609d8b4ddca23fa" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 35197 } } peers { permanent_uuid: "e688acdd9320482aa68b5a799632d805" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44821 } } peers { permanent_uuid: "739234e9520c416f92b38176137e07df" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45309 } }
I20260811 11:54:35.339790 4379 leader_election.cc:288] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7f5f7d65b4646eba609d8b4ddca23fa (127.3.221.67:35197), 739234e9520c416f92b38176137e07df (127.3.221.65:45309)
I20260811 11:54:35.345114 4120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:35.340116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.67:35197 (local address 127.3.221.66:39595)
0811 11:54:35.340258 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:35.340274 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:35.340422 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:35.340839 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:35.340843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:35.340892 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:35.341149 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:35.341166 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.342195 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.342199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:35.343337 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:35.343348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.344298 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.344302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:35.344436 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:35.344481 (+ 45us) client_negotiation.cc:769] Sending connection context
0811 11:54:35.344735 (+ 254us) client_negotiation.cc:240] Negotiation successful
0811 11:54:35.344933 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260811 11:54:35.345110 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:35.340258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:39595 (local address 127.3.221.67:35197)
0811 11:54:35.340431 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:35.340436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:35.340453 (+ 17us) server_negotiation.cc:407] Connection header received
0811 11:54:35.340520 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:35.340523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:35.340577 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:35.340672 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:35.341315 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.342003 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.343469 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.344091 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.344209 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:35.344257 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:35.344748 (+ 491us) server_negotiation.cc:299] Negotiation successful
0811 11:54:35.344837 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 11:54:35.346024 4380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:35.340118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:45309 (local address 127.3.221.66:60085)
0811 11:54:35.340677 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:35.340693 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:54:35.340791 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:54:35.341328 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:54:35.341331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:54:35.341360 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 11:54:35.341548 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:35.341556 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.342884 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.342888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:35.343959 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:54:35.343972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.345531 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.345536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:54:35.345667 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:35.345691 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 11:54:35.345762 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 11:54:35.345844 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":191,"thread_start_us":127,"threads_started":1}
I20260811 11:54:35.346038 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:35.340212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.221.66:60085 (local address 127.3.221.65:45309)
0811 11:54:35.340346 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:35.340351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:35.340837 (+ 486us) server_negotiation.cc:407] Connection header received
0811 11:54:35.340935 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:35.340939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:35.341019 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:35.341248 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 11:54:35.341758 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.342692 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.344189 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.345116 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.345365 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:35.345428 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:35.345811 (+ 383us) server_negotiation.cc:299] Negotiation successful
0811 11:54:35.345921 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 11:54:35.346477 4262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c308a73dcb4ca49fa6036663c0009e" candidate_uuid: "e688acdd9320482aa68b5a799632d805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f5f7d65b4646eba609d8b4ddca23fa" is_pre_election: true
I20260811 11:54:35.346899 4262 raft_consensus.cc:2454] T a9c308a73dcb4ca49fa6036663c0009e P e7f5f7d65b4646eba609d8b4ddca23fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e688acdd9320482aa68b5a799632d805 in term 0.
I20260811 11:54:35.346895 3805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c308a73dcb4ca49fa6036663c0009e" candidate_uuid: "e688acdd9320482aa68b5a799632d805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "739234e9520c416f92b38176137e07df" is_pre_election: true
I20260811 11:54:35.347325 3805 raft_consensus.cc:2454] T a9c308a73dcb4ca49fa6036663c0009e P 739234e9520c416f92b38176137e07df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e688acdd9320482aa68b5a799632d805 in term 0.
I20260811 11:54:35.347538 3913 leader_election.cc:302] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e688acdd9320482aa68b5a799632d805, e7f5f7d65b4646eba609d8b4ddca23fa; no voters:
I20260811 11:54:35.347859 4379 raft_consensus.cc:2781] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 11:54:35.347939 4379 raft_consensus.cc:482] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 11:54:35.347980 4379 raft_consensus.cc:3037] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:35.349082 4379 raft_consensus.cc:504] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f5f7d65b4646eba609d8b4ddca23fa" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 35197 } } peers { permanent_uuid: "e688acdd9320482aa68b5a799632d805" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44821 } } peers { permanent_uuid: "739234e9520c416f92b38176137e07df" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45309 } }
I20260811 11:54:35.349548 4379 leader_election.cc:288] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805 [CANDIDATE]: Term 1 election: Requested vote from peers e7f5f7d65b4646eba609d8b4ddca23fa (127.3.221.67:35197), 739234e9520c416f92b38176137e07df (127.3.221.65:45309)
I20260811 11:54:35.349980 4262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c308a73dcb4ca49fa6036663c0009e" candidate_uuid: "e688acdd9320482aa68b5a799632d805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f5f7d65b4646eba609d8b4ddca23fa"
I20260811 11:54:35.350111 4262 raft_consensus.cc:3037] T a9c308a73dcb4ca49fa6036663c0009e P e7f5f7d65b4646eba609d8b4ddca23fa [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:35.350188 3805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c308a73dcb4ca49fa6036663c0009e" candidate_uuid: "e688acdd9320482aa68b5a799632d805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "739234e9520c416f92b38176137e07df"
I20260811 11:54:35.350406 3805 raft_consensus.cc:3037] T a9c308a73dcb4ca49fa6036663c0009e P 739234e9520c416f92b38176137e07df [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:35.351265 4262 raft_consensus.cc:2454] T a9c308a73dcb4ca49fa6036663c0009e P e7f5f7d65b4646eba609d8b4ddca23fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e688acdd9320482aa68b5a799632d805 in term 1.
I20260811 11:54:35.351598 3913 leader_election.cc:302] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e688acdd9320482aa68b5a799632d805, e7f5f7d65b4646eba609d8b4ddca23fa; no voters:
I20260811 11:54:35.351847 4379 raft_consensus.cc:2781] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:35.351980 3805 raft_consensus.cc:2454] T a9c308a73dcb4ca49fa6036663c0009e P 739234e9520c416f92b38176137e07df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e688acdd9320482aa68b5a799632d805 in term 1.
I20260811 11:54:35.352234 4379 raft_consensus.cc:686] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805 [term 1 LEADER]: Becoming Leader. State: Replica: e688acdd9320482aa68b5a799632d805, State: Running, Role: LEADER
I20260811 11:54:35.352774 4379 consensus_queue.cc:237] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f5f7d65b4646eba609d8b4ddca23fa" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 35197 } } peers { permanent_uuid: "e688acdd9320482aa68b5a799632d805" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44821 } } peers { permanent_uuid: "739234e9520c416f92b38176137e07df" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45309 } }
W20260811 11:54:35.353447 3882 tablet.cc:2369] T a9c308a73dcb4ca49fa6036663c0009e P 739234e9520c416f92b38176137e07df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 11:54:35.357599 3560 catalog_manager.cc:5495] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805 reported cstate change: term changed from 0 to 1, leader changed from <none> to e688acdd9320482aa68b5a799632d805 (127.3.221.66). New cstate: current_term: 1 leader_uuid: "e688acdd9320482aa68b5a799632d805" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f5f7d65b4646eba609d8b4ddca23fa" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 35197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e688acdd9320482aa68b5a799632d805" member_type: VOTER last_known_addr { host: "127.3.221.66" port: 44821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "739234e9520c416f92b38176137e07df" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45309 } health_report { overall_health: UNKNOWN } } }
I20260811 11:54:35.391558 4361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:35.386460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38348 (local address 127.3.221.66:44821)
0811 11:54:35.386729 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:35.386734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:35.386758 (+ 24us) server_negotiation.cc:407] Connection header received
0811 11:54:35.386910 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:35.386913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:35.386980 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:35.387101 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:54:35.387474 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.388046 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.389358 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:35.389663 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:35.389769 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:35.390281 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:54:35.390985 (+ 704us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:54:35.391103 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:35.391311 (+ 208us) server_negotiation.cc:299] Negotiation successful
0811 11:54:35.391380 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260811 11:54:35.399500 4262 raft_consensus.cc:1261] T a9c308a73dcb4ca49fa6036663c0009e P e7f5f7d65b4646eba609d8b4ddca23fa [term 1 FOLLOWER]: Refusing update from remote peer e688acdd9320482aa68b5a799632d805: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:54:35.399502 3805 raft_consensus.cc:1261] T a9c308a73dcb4ca49fa6036663c0009e P 739234e9520c416f92b38176137e07df [term 1 FOLLOWER]: Refusing update from remote peer e688acdd9320482aa68b5a799632d805: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:54:35.400293 4381 consensus_queue.cc:1035] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805 [LEADER]: Connected to new peer: Peer: permanent_uuid: "739234e9520c416f92b38176137e07df" member_type: VOTER last_known_addr { host: "127.3.221.65" port: 45309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:54:35.400496 4379 consensus_queue.cc:1035] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f5f7d65b4646eba609d8b4ddca23fa" member_type: VOTER last_known_addr { host: "127.3.221.67" port: 35197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:54:35.411253 4390 mvcc.cc:205] Tried to move back new op lower bound from 7317296232027971584 to 7317296231850356736. Current Snapshot: MvccSnapshot[applied={T|T < 7317296232027971584 or (T in {7317296232027971584})}]
I20260811 11:54:35.418277 3560 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:54:35.418540 3560 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 11:54:35.421350 3560 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63007ea0d8524b908c15ce330b071f0e: Sending DeleteTablet for 3 replicas of tablet a9c308a73dcb4ca49fa6036663c0009e
I20260811 11:54:35.422272 3996 tablet_service.cc:1514] Processing DeleteTablet for tablet a9c308a73dcb4ca49fa6036663c0009e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:54:35 UTC) from {username='slave'} at 127.0.0.1:38346
I20260811 11:54:35.422344 3773 tablet_service.cc:1514] Processing DeleteTablet for tablet a9c308a73dcb4ca49fa6036663c0009e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:54:35 UTC) from {username='slave'} at 127.0.0.1:45614
I20260811 11:54:35.422344 4229 tablet_service.cc:1514] Processing DeleteTablet for tablet a9c308a73dcb4ca49fa6036663c0009e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 11:54:35 UTC) from {username='slave'} at 127.0.0.1:50704
I20260811 11:54:35.423094 4399 tablet_replica.cc:330] T a9c308a73dcb4ca49fa6036663c0009e P 739234e9520c416f92b38176137e07df: stopping tablet replica
I20260811 11:54:35.423552 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 3670
I20260811 11:54:35.431651 4398 tablet_replica.cc:330] T a9c308a73dcb4ca49fa6036663c0009e P e7f5f7d65b4646eba609d8b4ddca23fa: stopping tablet replica
I20260811 11:54:35.432160 4398 raft_consensus.cc:2229] T a9c308a73dcb4ca49fa6036663c0009e P e7f5f7d65b4646eba609d8b4ddca23fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 11:54:35.432653 4398 raft_consensus.cc:2258] T a9c308a73dcb4ca49fa6036663c0009e P e7f5f7d65b4646eba609d8b4ddca23fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:54:35.433395 4397 tablet_replica.cc:330] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805: stopping tablet replica
I20260811 11:54:35.433722 4397 raft_consensus.cc:2229] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805 [term 1 LEADER]: Raft consensus shutting down.
I20260811 11:54:35.434155 4397 raft_consensus.cc:2258] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 11:54:35.435564 4398 ts_tablet_manager.cc:1905] T a9c308a73dcb4ca49fa6036663c0009e P e7f5f7d65b4646eba609d8b4ddca23fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:54:35.435812 4397 ts_tablet_manager.cc:1905] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 11:54:35.438648 4398 ts_tablet_manager.cc:1918] T a9c308a73dcb4ca49fa6036663c0009e P e7f5f7d65b4646eba609d8b4ddca23fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:54:35.438779 4398 log.cc:1196] T a9c308a73dcb4ca49fa6036663c0009e P e7f5f7d65b4646eba609d8b4ddca23fa: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-2/wal/wals/a9c308a73dcb4ca49fa6036663c0009e
I20260811 11:54:35.438719 4397 ts_tablet_manager.cc:1918] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 11:54:35.439066 4397 log.cc:1196] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805: Deleting WAL directory at /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786449004114841-3957-0/minicluster-data/ts-1/wal/wals/a9c308a73dcb4ca49fa6036663c0009e
I20260811 11:54:35.439079 4398 ts_tablet_manager.cc:1939] T a9c308a73dcb4ca49fa6036663c0009e P e7f5f7d65b4646eba609d8b4ddca23fa: Deleting consensus metadata
I20260811 11:54:35.439550 4397 ts_tablet_manager.cc:1939] T a9c308a73dcb4ca49fa6036663c0009e P e688acdd9320482aa68b5a799632d805: Deleting consensus metadata
I20260811 11:54:35.440555 3541 catalog_manager.cc:4841] TS e7f5f7d65b4646eba609d8b4ddca23fa (127.3.221.67:35197): tablet a9c308a73dcb4ca49fa6036663c0009e (table test-table [id=e1cffe860bee447bbf7fb8a819ac3ac4]) successfully deleted
I20260811 11:54:35.440853 3543 catalog_manager.cc:4841] TS e688acdd9320482aa68b5a799632d805 (127.3.221.66:44821): tablet a9c308a73dcb4ca49fa6036663c0009e (table test-table [id=e1cffe860bee447bbf7fb8a819ac3ac4]) successfully deleted
W20260811 11:54:35.441926 3542 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.221.65:45309 (error 108)
I20260811 11:54:35.442113 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 3886
I20260811 11:54:35.451431 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:35.451047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.65:45309 (local address 127.0.0.1:45620)
0811 11:54:35.451228 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0811 11:54:35.451333 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.65:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260811 11:54:35.451644 3542 catalog_manager.cc:4584] TS 739234e9520c416f92b38176137e07df (127.3.221.65:45309): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9c308a73dcb4ca49fa6036663c0009e: Network error: Client connection negotiation failed: client connection to 127.3.221.65:45309: connect: Connection refused (error 111)
I20260811 11:54:35.460003 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 4124
I20260811 11:54:35.479768 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 3515
2026-08-11T11:54:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1409 ms)
[----------] 6 tests from EncryptionPolicyTest (11868 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:54:35 dist-test-slave-bndk krb5kdc[4406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:54:35 dist-test-slave-bndk krb5kdc[4406](info): set up 2 sockets
Aug 11 11:54:35 dist-test-slave-bndk krb5kdc[4406](info): commencing operation
krb5kdc: starting...
W20260811 11:54:36.669009 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:54:36 dist-test-slave-bndk krb5kdc[4406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:54:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:54:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:54:36.804376 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40007
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:36.924259 4431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:36.924559 4431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:36.924602 4431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:36.928763 4431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:54:36.928850 4431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:36.928869 4431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:36.928884 4431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:54:36.928898 4431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:54:36.934624 4431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40007
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:36.935784 4431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:36.936841 4431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:36.945484 4441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:36.945488 4438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:36.945678 4431 server_base.cc:1034] running on GCE node
W20260811 11:54:36.945509 4445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:36.946412 4431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:36.947642 4431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:36.948818 4431 hybrid_clock.cc:648] HybridClock initialized: now 1786449276948804 us; error 29 us; skew 500 ppm
Aug 11 11:54:36 dist-test-slave-bndk krb5kdc[4406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:36.951835 4431 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:54:36.953004 4431 webserver.cc:460] Webserver started at http://127.0.0.1:38839/ using document root <none> and password file <none>
I20260811 11:54:36.953685 4431 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:36.953771 4431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:36.954005 4431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:36.956223 4431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "66564ea0b2384867b549f94394ba3f11"
format_stamp: "Formatted at 2026-08-11 11:54:36 on dist-test-slave-bndk"
server_key: "10c9fc9260fbd21470e91196019471e1"
server_key_iv: "cb6c3b3f23d3913e8382774c27f1b3d9"
server_key_version: "encryptionkey@0"
I20260811 11:54:36.956838 4431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "66564ea0b2384867b549f94394ba3f11"
format_stamp: "Formatted at 2026-08-11 11:54:36 on dist-test-slave-bndk"
server_key: "10c9fc9260fbd21470e91196019471e1"
server_key_iv: "cb6c3b3f23d3913e8382774c27f1b3d9"
server_key_version: "encryptionkey@0"
I20260811 11:54:36.961001 4431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260811 11:54:36.963809 4456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:36.965929 4431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:54:36.966084 4431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "66564ea0b2384867b549f94394ba3f11"
format_stamp: "Formatted at 2026-08-11 11:54:36 on dist-test-slave-bndk"
server_key: "10c9fc9260fbd21470e91196019471e1"
server_key_iv: "cb6c3b3f23d3913e8382774c27f1b3d9"
server_key_version: "encryptionkey@0"
I20260811 11:54:36.966220 4431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:36.982254 4431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:36.985685 4431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:36.985893 4431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:36.993505 4431 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40007
I20260811 11:54:36.993521 4554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40007 every 8 connection(s)
I20260811 11:54:36.994561 4431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:54:36.997664 4560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:37.002548 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 4431
I20260811 11:54:37.002699 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:54:37.003021 3957 external_mini_cluster.cc:1428] Setting key 3ae3d6b84ad1f83e5ac33bbc2bbe5bcb
I20260811 11:54:37.004063 4560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11: Bootstrap starting.
I20260811 11:54:37.006598 4560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:37.007597 4560 log.cc:824] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:37.010224 4560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11: No bootstrap required, opened a new log
Aug 11 11:54:37 dist-test-slave-bndk krb5kdc[4406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 11:54:37.015599 4560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66564ea0b2384867b549f94394ba3f11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40007 } }
I20260811 11:54:37.015892 4560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:37.015954 4560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66564ea0b2384867b549f94394ba3f11, State: Initialized, Role: FOLLOWER
I20260811 11:54:37.016439 4560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66564ea0b2384867b549f94394ba3f11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40007 } }
I20260811 11:54:37.016589 4560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:54:37.016793 4560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:54:37.016939 4560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:37.017998 4560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66564ea0b2384867b549f94394ba3f11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40007 } }
I20260811 11:54:37.018373 4560 leader_election.cc:302] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66564ea0b2384867b549f94394ba3f11; no voters:
I20260811 11:54:37.018720 4560 leader_election.cc:288] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:54:37.018874 4567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:37.019097 4567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11 [term 1 LEADER]: Becoming Leader. State: Replica: 66564ea0b2384867b549f94394ba3f11, State: Running, Role: LEADER
I20260811 11:54:37.019562 4567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66564ea0b2384867b549f94394ba3f11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40007 } }
I20260811 11:54:37.020262 4560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:54:37.020272 4564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:37.004600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57783 (local address 127.0.0.1:40007)
0811 11:54:37.005040 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:37.005056 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:37.005095 (+ 39us) server_negotiation.cc:407] Connection header received
0811 11:54:37.005835 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:37.005855 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:37.006219 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:37.006515 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:37.007477 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:37.008831 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:37.010053 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:37.010606 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:37.010739 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:37.014596 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:37.014602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:37.014620 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:37.014659 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:37.016580 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:37.017324 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:37.017330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:37.017336 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:37.017470 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:37.017827 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:37.017830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:37.017831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:37.018331 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:37.018539 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:37.018915 (+ 376us) server_negotiation.cc:299] Negotiation successful
0811 11:54:37.019201 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":127,"threads_started":1}
I20260811 11:54:37.021378 4573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66564ea0b2384867b549f94394ba3f11. Latest consensus state: current_term: 1 leader_uuid: "66564ea0b2384867b549f94394ba3f11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66564ea0b2384867b549f94394ba3f11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40007 } } }
I20260811 11:54:37.021539 4573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:37.021865 4574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66564ea0b2384867b549f94394ba3f11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66564ea0b2384867b549f94394ba3f11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40007 } } }
I20260811 11:54:37.021963 4574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:37.023613 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260811 11:54:37.025789 4593 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:54:37.025877 4593 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:54:37.025946 4583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:54:37.026814 4583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:54:37.032800 4583 catalog_manager.cc:1347] Generated new cluster ID: a5431b37299049999a2d359983d7ee79
I20260811 11:54:37.032882 4583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:54:37.037585 4564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:37.025636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35401 (local address 127.0.0.1:40007)
0811 11:54:37.025747 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:37.025752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:37.029278 (+ 3526us) server_negotiation.cc:407] Connection header received
0811 11:54:37.029407 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:37.029410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:37.029462 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:37.029550 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:37.030447 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:37.031104 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:37.032297 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:37.032616 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:37.032722 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:37.034459 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:37.034462 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:37.034464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:37.034490 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:37.035823 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:37.036298 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:37.036300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:37.036302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:37.036384 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:37.036688 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:37.036690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:37.036692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:37.036868 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:37.037003 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:37.037311 (+ 308us) server_negotiation.cc:299] Negotiation successful
0811 11:54:37.037420 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 11:54:37.045367 4583 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:54:37.046358 4583 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:54:37.058221 4583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66564ea0b2384867b549f94394ba3f11: Generated new TSK 0
I20260811 11:54:37.059171 4583 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:54:37.067715 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 4431
W20260811 11:54:37.088579 4401 connection.cc:535] client connection to 127.0.0.1:40007 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40007): Connection reset by peer (error 104)
2026-08-11T11:54:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1588 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:54:37 dist-test-slave-bndk krb5kdc[4614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:54:37 dist-test-slave-bndk krb5kdc[4614](info): set up 2 sockets
Aug 11 11:54:37 dist-test-slave-bndk krb5kdc[4614](info): commencing operation
krb5kdc: starting...
W20260811 11:54:38.277793 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:54:38 dist-test-slave-bndk krb5kdc[4614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:54:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:54:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:54:38.420439 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39467
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:38.541802 4631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:38.542116 4631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:38.542177 4631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:38.546375 4631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:54:38.546459 4631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:38.546483 4631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:38.546506 4631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:54:38.546535 4631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:54:38.552346 4631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39467
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:38.553794 4631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:38.554942 4631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:38.564460 4646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:38.564464 4640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:38.564460 4639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:38.564899 4631 server_base.cc:1034] running on GCE node
I20260811 11:54:38.565579 4631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:38.566810 4631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:38.568003 4631 hybrid_clock.cc:648] HybridClock initialized: now 1786449278567982 us; error 38 us; skew 500 ppm
Aug 11 11:54:38 dist-test-slave-bndk krb5kdc[4614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:38.571086 4631 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:54:38.572105 4631 webserver.cc:460] Webserver started at http://127.0.0.1:43343/ using document root <none> and password file <none>
I20260811 11:54:38.572813 4631 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:38.572911 4631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:38.573141 4631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:38.575491 4631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "5a408c3d3e6a4215a2203e82231dfa85"
format_stamp: "Formatted at 2026-08-11 11:54:38 on dist-test-slave-bndk"
server_key: "471de4a6d12989c6f52858cc4239a4a0"
server_key_iv: "e914b206356c0a5f535e4b62fc59053c"
server_key_version: "encryptionkey@0"
I20260811 11:54:38.576136 4631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "5a408c3d3e6a4215a2203e82231dfa85"
format_stamp: "Formatted at 2026-08-11 11:54:38 on dist-test-slave-bndk"
server_key: "471de4a6d12989c6f52858cc4239a4a0"
server_key_iv: "e914b206356c0a5f535e4b62fc59053c"
server_key_version: "encryptionkey@0"
I20260811 11:54:38.581012 4631 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260811 11:54:38.584100 4653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:38.585738 4631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:54:38.585888 4631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "5a408c3d3e6a4215a2203e82231dfa85"
format_stamp: "Formatted at 2026-08-11 11:54:38 on dist-test-slave-bndk"
server_key: "471de4a6d12989c6f52858cc4239a4a0"
server_key_iv: "e914b206356c0a5f535e4b62fc59053c"
server_key_version: "encryptionkey@0"
I20260811 11:54:38.586023 4631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:38.603569 4631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:38.607182 4631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:38.607484 4631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:38.617072 4631 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39467
I20260811 11:54:38.617127 4738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39467 every 8 connection(s)
I20260811 11:54:38.618213 4631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:54:38.618846 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 4631
I20260811 11:54:38.619001 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:54:38.619305 3957 external_mini_cluster.cc:1428] Setting key 6d37ce8cfb03a3ecdf0272e668138e8a
I20260811 11:54:38.622052 4739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:38.629199 4739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85: Bootstrap starting.
I20260811 11:54:38.630515 4748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:38.621165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36249 (local address 127.0.0.1:39467)
0811 11:54:38.621682 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:38.621698 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:38.621737 (+ 39us) server_negotiation.cc:407] Connection header received
0811 11:54:38.622454 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:38.622474 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:38.622841 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:38.623156 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:38.623543 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:38.624600 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:38.625882 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:38.626530 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:38.626679 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:38.627737 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:38.627742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:38.627757 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:38.628187 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:38.628253 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:38.628280 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:38.628415 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:38.628938 (+ 523us) server_negotiation.cc:299] Negotiation successful
0811 11:54:38.629196 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":159,"threads_started":1}
I20260811 11:54:38.632169 4739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:38.633101 4739 log.cc:824] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:38.634492 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:54:38.636277 4739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85: No bootstrap required, opened a new log
I20260811 11:54:38.641443 4748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:38.636624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44741 (local address 127.0.0.1:39467)
0811 11:54:38.636761 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:38.636766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:38.636954 (+ 188us) server_negotiation.cc:407] Connection header received
0811 11:54:38.637055 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:38.637058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:38.637107 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:38.637199 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:38.637480 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:38.638243 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:38.639293 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:38.639750 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:38.639871 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:38.640751 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:38.640755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:38.640758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:38.640773 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:38.640817 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:38.640846 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:38.640950 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:38.641121 (+ 171us) server_negotiation.cc:299] Negotiation successful
0811 11:54:38.641169 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 11:54:38.642086 4739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a408c3d3e6a4215a2203e82231dfa85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39467 } }
I20260811 11:54:38.642359 4739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:38.642396 4739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a408c3d3e6a4215a2203e82231dfa85, State: Initialized, Role: FOLLOWER
I20260811 11:54:38.642875 4739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a408c3d3e6a4215a2203e82231dfa85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39467 } }
I20260811 11:54:38.643016 4739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:54:38.643082 4739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:54:38.643186 4739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:38.644230 4739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a408c3d3e6a4215a2203e82231dfa85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39467 } }
I20260811 11:54:38.644606 4739 leader_election.cc:302] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a408c3d3e6a4215a2203e82231dfa85; no voters:
I20260811 11:54:38.644965 4739 leader_election.cc:288] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:54:38.645109 4759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:38.645388 4759 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85 [term 1 LEADER]: Becoming Leader. State: Replica: 5a408c3d3e6a4215a2203e82231dfa85, State: Running, Role: LEADER
I20260811 11:54:38.645762 4759 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a408c3d3e6a4215a2203e82231dfa85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39467 } }
I20260811 11:54:38.646361 4739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:54:38.647542 4764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a408c3d3e6a4215a2203e82231dfa85. Latest consensus state: current_term: 1 leader_uuid: "5a408c3d3e6a4215a2203e82231dfa85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a408c3d3e6a4215a2203e82231dfa85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39467 } } }
I20260811 11:54:38.647670 4764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:38.648075 4772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:54:38.648075 4763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a408c3d3e6a4215a2203e82231dfa85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a408c3d3e6a4215a2203e82231dfa85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39467 } } }
I20260811 11:54:38.648211 4763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:38.651993 4772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:54:38.658752 4772 catalog_manager.cc:1347] Generated new cluster ID: cbea449a41884e53bb0e939a4240f4e1
I20260811 11:54:38.658861 4772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:54:38.673207 4772 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:54:38.674319 4772 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:54:38.679827 4772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a408c3d3e6a4215a2203e82231dfa85: Generated new TSK 0
I20260811 11:54:38.680737 4772 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:54:38.694394 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 4631
W20260811 11:54:38.714111 4612 connection.cc:535] client connection to 127.0.0.1:39467 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39467): Connection reset by peer (error 104)
2026-08-11T11:54:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1625 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:54:38 dist-test-slave-bndk krb5kdc[4793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:54:38 dist-test-slave-bndk krb5kdc[4793](info): set up 2 sockets
Aug 11 11:54:38 dist-test-slave-bndk krb5kdc[4793](info): commencing operation
krb5kdc: starting...
W20260811 11:54:39.897540 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:54:39 dist-test-slave-bndk krb5kdc[4793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:54:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:54:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:54:40.036136 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42821
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:40.165868 4815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:40.166507 4815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:40.166585 4815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:40.171311 4815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:54:40.171420 4815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:40.171449 4815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:40.171480 4815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:54:40.171502 4815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:54:40.177510 4815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42821
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:40.178769 4815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:40.179875 4815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:40.189589 4826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:40.189615 4825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:40.189668 4815 server_base.cc:1034] running on GCE node
W20260811 11:54:40.189617 4831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:40.190474 4815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:40.191732 4815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:40.193126 4815 hybrid_clock.cc:648] HybridClock initialized: now 1786449280193099 us; error 66 us; skew 500 ppm
Aug 11 11:54:40 dist-test-slave-bndk krb5kdc[4793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:40.202474 4815 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:54:40.203564 4815 webserver.cc:460] Webserver started at http://127.0.0.1:41051/ using document root <none> and password file <none>
I20260811 11:54:40.204226 4815 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:40.204319 4815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:40.204552 4815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:40.206907 4815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "6b3c0c835d3048c7ab3f2c3e8bb95c91"
format_stamp: "Formatted at 2026-08-11 11:54:40 on dist-test-slave-bndk"
server_key: "d23713495d3d9ca6b1f6a6756437829d"
server_key_iv: "6202b642c8603feb6358e0997e688748"
server_key_version: "encryptionkey@0"
I20260811 11:54:40.207542 4815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "6b3c0c835d3048c7ab3f2c3e8bb95c91"
format_stamp: "Formatted at 2026-08-11 11:54:40 on dist-test-slave-bndk"
server_key: "d23713495d3d9ca6b1f6a6756437829d"
server_key_iv: "6202b642c8603feb6358e0997e688748"
server_key_version: "encryptionkey@0"
I20260811 11:54:40.211810 4815 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 11:54:40.214859 4840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:40.216332 4815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 11:54:40.216481 4815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "6b3c0c835d3048c7ab3f2c3e8bb95c91"
format_stamp: "Formatted at 2026-08-11 11:54:40 on dist-test-slave-bndk"
server_key: "d23713495d3d9ca6b1f6a6756437829d"
server_key_iv: "6202b642c8603feb6358e0997e688748"
server_key_version: "encryptionkey@0"
I20260811 11:54:40.216614 4815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:40.222098 4815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:40.225989 4815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:40.226207 4815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:40.234540 4815 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42821
I20260811 11:54:40.234613 4929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42821 every 8 connection(s)
I20260811 11:54:40.235608 4815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:54:40.238775 4930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:40.245124 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 4815
I20260811 11:54:40.245334 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:54:40.245661 3957 external_mini_cluster.cc:1428] Setting key f81d39637717b68c9bdc8c5f4e1da8b7
I20260811 11:54:40.245721 4930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91: Bootstrap starting.
I20260811 11:54:40.248539 4930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:40.249411 4930 log.cc:824] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:40.251566 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:40.247228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50857 (local address 127.0.0.1:42821)
0811 11:54:40.247759 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:40.247775 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:40.247816 (+ 41us) server_negotiation.cc:407] Connection header received
0811 11:54:40.248541 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:40.248562 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:40.248920 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:40.249320 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:40.250218 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:40.250224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:40.250242 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:40.250255 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:40.250319 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:40.250341 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:40.250437 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:40.250573 (+ 136us) server_negotiation.cc:299] Negotiation successful
0811 11:54:40.250737 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":163,"threads_started":1}
I20260811 11:54:40.251765 4930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91: No bootstrap required, opened a new log
I20260811 11:54:40.255198 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:54:40.256222 4930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3c0c835d3048c7ab3f2c3e8bb95c91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42821 } }
I20260811 11:54:40.256557 4930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:40.256657 4930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b3c0c835d3048c7ab3f2c3e8bb95c91, State: Initialized, Role: FOLLOWER
I20260811 11:54:40.257189 4930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3c0c835d3048c7ab3f2c3e8bb95c91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42821 } }
I20260811 11:54:40.257405 4930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:54:40.257481 4930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:54:40.257591 4930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:40.258858 4930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3c0c835d3048c7ab3f2c3e8bb95c91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42821 } }
I20260811 11:54:40.259281 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:40.257668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49699 (local address 127.0.0.1:42821)
0811 11:54:40.257790 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:40.257795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:40.257994 (+ 199us) server_negotiation.cc:407] Connection header received
0811 11:54:40.258103 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:40.258106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:40.258146 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:40.258213 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:40.258892 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:40.258896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:40.258899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:40.258911 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:40.258944 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:40.258968 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:40.259056 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:40.259123 (+ 67us) server_negotiation.cc:299] Negotiation successful
0811 11:54:40.259161 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260811 11:54:40.259274 4930 leader_election.cc:302] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b3c0c835d3048c7ab3f2c3e8bb95c91; no voters:
I20260811 11:54:40.259653 4930 leader_election.cc:288] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:54:40.259729 4946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:40.259939 4946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91 [term 1 LEADER]: Becoming Leader. State: Replica: 6b3c0c835d3048c7ab3f2c3e8bb95c91, State: Running, Role: LEADER
I20260811 11:54:40.260311 4946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3c0c835d3048c7ab3f2c3e8bb95c91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42821 } }
I20260811 11:54:40.261092 4930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:54:40.261751 4946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b3c0c835d3048c7ab3f2c3e8bb95c91. Latest consensus state: current_term: 1 leader_uuid: "6b3c0c835d3048c7ab3f2c3e8bb95c91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3c0c835d3048c7ab3f2c3e8bb95c91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42821 } } }
I20260811 11:54:40.261962 4946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:40.262151 4948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b3c0c835d3048c7ab3f2c3e8bb95c91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3c0c835d3048c7ab3f2c3e8bb95c91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42821 } } }
I20260811 11:54:40.262297 4948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:40.262470 4957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:54:40.265879 4957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:54:40.272199 4957 catalog_manager.cc:1347] Generated new cluster ID: f172560d73604b8b98237c4b73682550
I20260811 11:54:40.272285 4957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:54:40.280587 4957 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:54:40.280932 4957 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:54:40.291781 4957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b3c0c835d3048c7ab3f2c3e8bb95c91: Generated new TSK 0
I20260811 11:54:40.292716 4957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:54:40.297398 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 4815
2026-08-11T11:54:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1599 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:54:40 dist-test-slave-bndk krb5kdc[4984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:54:40 dist-test-slave-bndk krb5kdc[4984](info): set up 2 sockets
Aug 11 11:54:40 dist-test-slave-bndk krb5kdc[4984](info): commencing operation
krb5kdc: starting...
W20260811 11:54:41.488690 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:54:41 dist-test-slave-bndk krb5kdc[4984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:54:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:54:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:54:41.627998 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:41.755554 5009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:41.755908 5009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:41.755972 5009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:41.760217 5009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:54:41.760313 5009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:41.760344 5009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:41.760365 5009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:54:41.760386 5009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:54:41.766201 5009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:41.767419 5009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:41.768473 5009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:41.779278 5021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:41.779279 5017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:41.779279 5018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:41.779495 5009 server_base.cc:1034] running on GCE node
I20260811 11:54:41.780112 5009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:41.781378 5009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:41.782558 5009 hybrid_clock.cc:648] HybridClock initialized: now 1786449281782541 us; error 33 us; skew 500 ppm
Aug 11 11:54:41 dist-test-slave-bndk krb5kdc[4984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:41.785616 5009 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:54:41.786656 5009 webserver.cc:460] Webserver started at http://127.0.0.1:32861/ using document root <none> and password file <none>
I20260811 11:54:41.787236 5009 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:41.787314 5009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:41.787533 5009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:41.789755 5009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "2c74d16464094aa1ab7d281dd4932b80"
format_stamp: "Formatted at 2026-08-11 11:54:41 on dist-test-slave-bndk"
server_key: "64e59f30c9125a3beecbd471c3386ef6"
server_key_iv: "25c6d02f63fc9b6e51e5a7149c88fd73"
server_key_version: "encryptionkey@0"
I20260811 11:54:41.790355 5009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "2c74d16464094aa1ab7d281dd4932b80"
format_stamp: "Formatted at 2026-08-11 11:54:41 on dist-test-slave-bndk"
server_key: "64e59f30c9125a3beecbd471c3386ef6"
server_key_iv: "25c6d02f63fc9b6e51e5a7149c88fd73"
server_key_version: "encryptionkey@0"
I20260811 11:54:41.794420 5009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 11:54:41.797266 5033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:41.798694 5009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 11:54:41.798856 5009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "2c74d16464094aa1ab7d281dd4932b80"
format_stamp: "Formatted at 2026-08-11 11:54:41 on dist-test-slave-bndk"
server_key: "64e59f30c9125a3beecbd471c3386ef6"
server_key_iv: "25c6d02f63fc9b6e51e5a7149c88fd73"
server_key_version: "encryptionkey@0"
I20260811 11:54:41.798988 5009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:41.813284 5009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:41.818351 5009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:41.818630 5009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:41.826862 5009 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37127
I20260811 11:54:41.826917 5126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37127 every 8 connection(s)
I20260811 11:54:41.827930 5009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:54:41.831123 5128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:41.837175 5128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80: Bootstrap starting.
I20260811 11:54:41.837205 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 5009
I20260811 11:54:41.837416 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:54:41.837735 3957 external_mini_cluster.cc:1428] Setting key 4ecfb51ae3387011c4e1fe5be91244dc
I20260811 11:54:41.840085 5128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:41.840914 5128 log.cc:824] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:41.843020 5128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80: No bootstrap required, opened a new log
I20260811 11:54:41.846895 5128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c74d16464094aa1ab7d281dd4932b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } }
I20260811 11:54:41.847162 5128 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:41.847221 5128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c74d16464094aa1ab7d281dd4932b80, State: Initialized, Role: FOLLOWER
Aug 11 11:54:41 dist-test-slave-bndk krb5kdc[4984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 11:54:41.847723 5128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c74d16464094aa1ab7d281dd4932b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } }
I20260811 11:54:41.847901 5128 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:54:41.847980 5128 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:54:41.848088 5128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:41.849364 5128 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c74d16464094aa1ab7d281dd4932b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } }
I20260811 11:54:41.849776 5128 leader_election.cc:302] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c74d16464094aa1ab7d281dd4932b80; no voters:
I20260811 11:54:41.850185 5128 leader_election.cc:288] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:54:41.850308 5134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:41.850575 5134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80 [term 1 LEADER]: Becoming Leader. State: Replica: 2c74d16464094aa1ab7d281dd4932b80, State: Running, Role: LEADER
I20260811 11:54:41.850994 5134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c74d16464094aa1ab7d281dd4932b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } }
I20260811 11:54:41.851622 5128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:54:41.853591 5132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:41.839553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50707 (local address 127.0.0.1:37127)
0811 11:54:41.840219 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:41.840241 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:41.840293 (+ 52us) server_negotiation.cc:407] Connection header received
0811 11:54:41.841012 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:41.841030 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:41.841484 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:41.841836 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:41.842719 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:41.843772 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:41.844657 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:41.845163 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:41.845371 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:41.847928 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:41.847935 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:41.847954 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:41.847991 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:41.850105 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:41.850707 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:41.850712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:41.850719 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:41.850866 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:41.851209 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:41.851227 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:41.851229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:41.851773 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:41.851970 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:41.852309 (+ 339us) server_negotiation.cc:299] Negotiation successful
0811 11:54:41.852644 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":183,"threads_started":1}
I20260811 11:54:41.855957 5135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c74d16464094aa1ab7d281dd4932b80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c74d16464094aa1ab7d281dd4932b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } } }
I20260811 11:54:41.855957 5136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c74d16464094aa1ab7d281dd4932b80. Latest consensus state: current_term: 1 leader_uuid: "2c74d16464094aa1ab7d281dd4932b80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c74d16464094aa1ab7d281dd4932b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } } }
I20260811 11:54:41.856182 5136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:41.856182 5135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:41.858033 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260811 11:54:41.858319 5153 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:54:41.858407 5153 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:54:41.858492 5157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:54:41.859448 5157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:54:41.866123 5157 catalog_manager.cc:1347] Generated new cluster ID: c12e7a299ad1431bbf1dc5f9d6669b21
I20260811 11:54:41.866297 5157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:54:41.870600 5132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:41.860402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44179 (local address 127.0.0.1:37127)
0811 11:54:41.860540 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:41.860544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:41.861546 (+ 1002us) server_negotiation.cc:407] Connection header received
0811 11:54:41.861744 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:41.861748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:41.861804 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:41.861902 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:41.862749 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:41.863563 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:41.864755 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:41.865100 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:41.865212 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:41.866925 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:41.866929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:41.866932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:41.866958 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:41.868572 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:41.869115 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:41.869118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:41.869119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:41.869199 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:41.869529 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:41.869534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:41.869535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:41.869906 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:41.870051 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:41.870254 (+ 203us) server_negotiation.cc:299] Negotiation successful
0811 11:54:41.870403 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 11:54:41.878605 5157 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:54:41.879951 5157 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:54:41.899341 5157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c74d16464094aa1ab7d281dd4932b80: Generated new TSK 0
I20260811 11:54:41.900230 5157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:54:41.909885 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 5009
2026-08-11T11:54:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1617 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:54:41 dist-test-slave-bndk krb5kdc[5175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:54:41 dist-test-slave-bndk krb5kdc[5175](info): set up 2 sockets
Aug 11 11:54:41 dist-test-slave-bndk krb5kdc[5175](info): commencing operation
krb5kdc: starting...
W20260811 11:54:43.106612 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:54:43 dist-test-slave-bndk krb5kdc[5175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:54:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:54:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:54:43.240417 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40963
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:43.361600 5197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:43.361940 5197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:43.362003 5197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:43.366221 5197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:54:43.366308 5197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:43.366338 5197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:43.366359 5197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:54:43.366380 5197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:54:43.372155 5197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:43.373389 5197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:43.374455 5197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:43.383754 5203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:43.383684 5204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:43.383740 5206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:43.383713 5197 server_base.cc:1034] running on GCE node
I20260811 11:54:43.384799 5197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:43.386101 5197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:43.387279 5197 hybrid_clock.cc:648] HybridClock initialized: now 1786449283387259 us; error 39 us; skew 500 ppm
Aug 11 11:54:43 dist-test-slave-bndk krb5kdc[5175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:43.390345 5197 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:54:43.391360 5197 webserver.cc:460] Webserver started at http://127.0.0.1:34287/ using document root <none> and password file <none>
I20260811 11:54:43.391948 5197 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:43.392020 5197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:43.392256 5197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:43.394482 5197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "e4c3f39111624046a236a24bec8bff83"
format_stamp: "Formatted at 2026-08-11 11:54:43 on dist-test-slave-bndk"
server_key: "29ef06933d9460b7ff1043821b03b390"
server_key_iv: "cbdf2a8a1612c53fd6c2b9b798f03a3a"
server_key_version: "encryptionkey@0"
I20260811 11:54:43.395077 5197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "e4c3f39111624046a236a24bec8bff83"
format_stamp: "Formatted at 2026-08-11 11:54:43 on dist-test-slave-bndk"
server_key: "29ef06933d9460b7ff1043821b03b390"
server_key_iv: "cbdf2a8a1612c53fd6c2b9b798f03a3a"
server_key_version: "encryptionkey@0"
I20260811 11:54:43.399072 5197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 11:54:43.402083 5215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:43.403522 5197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:54:43.403674 5197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "e4c3f39111624046a236a24bec8bff83"
format_stamp: "Formatted at 2026-08-11 11:54:43 on dist-test-slave-bndk"
server_key: "29ef06933d9460b7ff1043821b03b390"
server_key_iv: "cbdf2a8a1612c53fd6c2b9b798f03a3a"
server_key_version: "encryptionkey@0"
I20260811 11:54:43.403806 5197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:43.426558 5197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:43.430001 5197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:43.430210 5197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:43.437915 5197 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40963
I20260811 11:54:43.437942 5308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40963 every 8 connection(s)
I20260811 11:54:43.438980 5197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:54:43.441980 5310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:43.448133 5310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83: Bootstrap starting.
I20260811 11:54:43.448902 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 5197
I20260811 11:54:43.449051 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:54:43.449386 3957 external_mini_cluster.cc:1428] Setting key 03c52cb917be4a9dd53a69a8312999ba
I20260811 11:54:43.451081 5310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:43.451853 5310 log.cc:824] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:43.454006 5310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83: No bootstrap required, opened a new log
I20260811 11:54:43.458321 5315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:43.450610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51523 (local address 127.0.0.1:40963)
0811 11:54:43.451047 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:43.451064 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:43.451102 (+ 38us) server_negotiation.cc:407] Connection header received
0811 11:54:43.451802 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:43.451829 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:43.452212 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:43.452604 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:43.452987 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:43.454013 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:43.454880 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:43.455354 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:43.455510 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:43.456468 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:43.456474 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:43.456490 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:43.456922 (+ 432us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:43.456985 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:43.457011 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:43.457105 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:43.457294 (+ 189us) server_negotiation.cc:299] Negotiation successful
0811 11:54:43.457521 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":133,"threads_started":1}
I20260811 11:54:43.458307 5310 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c3f39111624046a236a24bec8bff83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40963 } }
I20260811 11:54:43.458603 5310 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:43.458663 5310 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c3f39111624046a236a24bec8bff83, State: Initialized, Role: FOLLOWER
I20260811 11:54:43.459143 5310 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c3f39111624046a236a24bec8bff83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40963 } }
I20260811 11:54:43.459283 5310 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:54:43.459369 5310 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:54:43.459477 5310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:43.460556 5310 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c3f39111624046a236a24bec8bff83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40963 } }
I20260811 11:54:43.460908 5310 leader_election.cc:302] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4c3f39111624046a236a24bec8bff83; no voters:
I20260811 11:54:43.461268 5310 leader_election.cc:288] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:54:43.461277 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:54:43.461354 5318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:43.461534 5318 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83 [term 1 LEADER]: Becoming Leader. State: Replica: e4c3f39111624046a236a24bec8bff83, State: Running, Role: LEADER
I20260811 11:54:43.461902 5318 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c3f39111624046a236a24bec8bff83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40963 } }
I20260811 11:54:43.462631 5310 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:54:43.463758 5322 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4c3f39111624046a236a24bec8bff83. Latest consensus state: current_term: 1 leader_uuid: "e4c3f39111624046a236a24bec8bff83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c3f39111624046a236a24bec8bff83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40963 } } }
I20260811 11:54:43.463892 5322 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:43.463727 5325 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4c3f39111624046a236a24bec8bff83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c3f39111624046a236a24bec8bff83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40963 } } }
I20260811 11:54:43.464140 5325 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83 [sys.catalog]: This master's current role is: LEADER
W20260811 11:54:43.467967 5349 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:54:43.468094 5349 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:54:43.468209 5340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:54:43.468361 5315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:43.464216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44935 (local address 127.0.0.1:40963)
0811 11:54:43.464344 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:43.464349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:43.464500 (+ 151us) server_negotiation.cc:407] Connection header received
0811 11:54:43.464694 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:43.464697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:43.464752 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:43.464839 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:43.465258 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:43.465768 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:43.466728 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:43.467011 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:43.467071 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:43.467941 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:43.467944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:43.467947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:43.467961 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:43.467989 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:43.467999 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:43.468058 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:43.468154 (+ 96us) server_negotiation.cc:299] Negotiation successful
0811 11:54:43.468211 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 11:54:43.469106 5340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:54:43.475690 5340 catalog_manager.cc:1347] Generated new cluster ID: 9e6f4c51fc1540049712c873ae505904
I20260811 11:54:43.475783 5340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:54:43.490327 5340 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:54:43.491326 5340 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:54:43.501667 5340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4c3f39111624046a236a24bec8bff83: Generated new TSK 0
I20260811 11:54:43.502651 5340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:54:43.508143 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 5197
2026-08-11T11:54:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1591 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:54:43 dist-test-slave-bndk krb5kdc[5358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:54:43 dist-test-slave-bndk krb5kdc[5358](info): set up 2 sockets
Aug 11 11:54:43 dist-test-slave-bndk krb5kdc[5358](info): commencing operation
krb5kdc: starting...
W20260811 11:54:44.684535 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:54:44 dist-test-slave-bndk krb5kdc[5358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:54:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:54:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:54:44.831040 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:44.956395 5373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:44.956840 5373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:44.956947 5373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:44.961297 5373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:54:44.961376 5373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:44.961397 5373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:44.961413 5373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:54:44.961427 5373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:54:44.967824 5373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:44.969085 5373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:44.970193 5373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:44.979579 5379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:44.979579 5382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:44.979579 5380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:44.979948 5373 server_base.cc:1034] running on GCE node
I20260811 11:54:44.980657 5373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:44.981884 5373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:44.983081 5373 hybrid_clock.cc:648] HybridClock initialized: now 1786449284983053 us; error 40 us; skew 500 ppm
Aug 11 11:54:44 dist-test-slave-bndk krb5kdc[5358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:44.986088 5373 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:54:44.987154 5373 webserver.cc:460] Webserver started at http://127.0.0.1:44785/ using document root <none> and password file <none>
I20260811 11:54:44.987740 5373 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:44.987823 5373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:44.988044 5373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:44.990334 5373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "43f29afd110940e895fa436c9b368eee"
format_stamp: "Formatted at 2026-08-11 11:54:44 on dist-test-slave-bndk"
server_key: "fe97c663144063441b325b7de17e8a7f"
server_key_iv: "0e429a2ec0f8a2c7642591f849b03100"
server_key_version: "encryptionkey@0"
I20260811 11:54:44.990919 5373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "43f29afd110940e895fa436c9b368eee"
format_stamp: "Formatted at 2026-08-11 11:54:44 on dist-test-slave-bndk"
server_key: "fe97c663144063441b325b7de17e8a7f"
server_key_iv: "0e429a2ec0f8a2c7642591f849b03100"
server_key_version: "encryptionkey@0"
I20260811 11:54:44.995332 5373 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 11:54:44.998082 5394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:44.999470 5373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:54:44.999610 5373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "43f29afd110940e895fa436c9b368eee"
format_stamp: "Formatted at 2026-08-11 11:54:44 on dist-test-slave-bndk"
server_key: "fe97c663144063441b325b7de17e8a7f"
server_key_iv: "0e429a2ec0f8a2c7642591f849b03100"
server_key_version: "encryptionkey@0"
I20260811 11:54:44.999727 5373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:45.004731 5373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:45.008134 5373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:45.008348 5373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:45.015904 5373 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45705
I20260811 11:54:45.015920 5477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45705 every 8 connection(s)
I20260811 11:54:45.016909 5373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:54:45.019850 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 5373
I20260811 11:54:45.019979 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:54:45.020022 5478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:45.020262 3957 external_mini_cluster.cc:1428] Setting key d4bdec493e6a496e31187157cb54a055
I20260811 11:54:45.026041 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:45.021743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49359 (local address 127.0.0.1:45705)
0811 11:54:45.022220 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:45.022239 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:45.022283 (+ 44us) server_negotiation.cc:407] Connection header received
0811 11:54:45.022964 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:45.022989 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:45.023376 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:45.023780 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:45.024540 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:45.024546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:45.024561 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:45.024573 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:45.024639 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:45.024666 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:45.024815 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:45.024899 (+ 84us) server_negotiation.cc:299] Negotiation successful
0811 11:54:45.025055 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":133,"threads_started":1}
I20260811 11:54:45.027859 5478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee: Bootstrap starting.
I20260811 11:54:45.029459 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:54:45.030762 5478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:45.031711 5478 log.cc:824] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:45.033370 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:45.031572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55299 (local address 127.0.0.1:45705)
0811 11:54:45.031708 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:45.031713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:45.031917 (+ 204us) server_negotiation.cc:407] Connection header received
0811 11:54:45.032033 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:45.032037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:45.032079 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:45.032165 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:45.032895 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:45.032899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:45.032901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:45.032913 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:45.032952 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:45.032965 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:45.033056 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:45.033178 (+ 122us) server_negotiation.cc:299] Negotiation successful
0811 11:54:45.033242 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 11:54:45.034256 5478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee: No bootstrap required, opened a new log
I20260811 11:54:45.038625 5478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f29afd110940e895fa436c9b368eee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45705 } }
I20260811 11:54:45.038913 5478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:45.038982 5478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f29afd110940e895fa436c9b368eee, State: Initialized, Role: FOLLOWER
I20260811 11:54:45.039455 5478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f29afd110940e895fa436c9b368eee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45705 } }
I20260811 11:54:45.039603 5478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:54:45.039673 5478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:54:45.039778 5478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:45.040817 5478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f29afd110940e895fa436c9b368eee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45705 } }
I20260811 11:54:45.041201 5478 leader_election.cc:302] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43f29afd110940e895fa436c9b368eee; no voters:
I20260811 11:54:45.041591 5478 leader_election.cc:288] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:54:45.041728 5499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:45.041973 5499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee [term 1 LEADER]: Becoming Leader. State: Replica: 43f29afd110940e895fa436c9b368eee, State: Running, Role: LEADER
I20260811 11:54:45.042289 5499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f29afd110940e895fa436c9b368eee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45705 } }
I20260811 11:54:45.042932 5478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:54:45.044445 5502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43f29afd110940e895fa436c9b368eee. Latest consensus state: current_term: 1 leader_uuid: "43f29afd110940e895fa436c9b368eee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f29afd110940e895fa436c9b368eee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45705 } } }
I20260811 11:54:45.044595 5502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:45.045533 5501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43f29afd110940e895fa436c9b368eee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f29afd110940e895fa436c9b368eee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45705 } } }
I20260811 11:54:45.045675 5501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee [sys.catalog]: This master's current role is: LEADER
W20260811 11:54:45.048033 5522 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:54:45.048110 5522 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:54:45.048204 5518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:54:45.048934 5518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:54:45.055143 5518 catalog_manager.cc:1347] Generated new cluster ID: 5e9ea834718d47bc8d96d3cd4c799fe0
I20260811 11:54:45.055227 5518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:54:45.067849 5518 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:54:45.068260 5518 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:54:45.083997 5518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43f29afd110940e895fa436c9b368eee: Generated new TSK 0
I20260811 11:54:45.085011 5518 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:54:45.093480 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 5373
2026-08-11T11:54:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1592 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:54:45 dist-test-slave-bndk krb5kdc[5531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:54:45 dist-test-slave-bndk krb5kdc[5531](info): set up 2 sockets
Aug 11 11:54:45 dist-test-slave-bndk krb5kdc[5531](info): commencing operation
krb5kdc: starting...
W20260811 11:54:46.280115 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:54:46 dist-test-slave-bndk krb5kdc[5531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:54:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:54:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:54:46.418210 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40021
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:46.546152 5553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:46.546483 5553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:46.546550 5553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:46.550773 5553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:54:46.550843 5553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:46.550871 5553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:46.550894 5553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:54:46.550921 5553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:54:46.556595 5553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:46.557796 5553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:46.558851 5553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:46.568146 5563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:46.568171 5564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:46.568253 5553 server_base.cc:1034] running on GCE node
W20260811 11:54:46.568171 5567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:46.569794 5553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:46.571396 5553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:46.572610 5553 hybrid_clock.cc:648] HybridClock initialized: now 1786449286572581 us; error 42 us; skew 500 ppm
Aug 11 11:54:46 dist-test-slave-bndk krb5kdc[5531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:46.585070 5553 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:54:46.586238 5553 webserver.cc:460] Webserver started at http://127.0.0.1:33079/ using document root <none> and password file <none>
I20260811 11:54:46.586853 5553 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:46.586920 5553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:46.587128 5553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:46.589483 5553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "81d4bc41d1984bca82ffec9ac1296056"
format_stamp: "Formatted at 2026-08-11 11:54:46 on dist-test-slave-bndk"
server_key: "875d49bb0ecf92fbb721344eb31d44b4"
server_key_iv: "4534fa8bb2b7e6ac34b8dc8c53ea54a3"
server_key_version: "encryptionkey@0"
I20260811 11:54:46.590099 5553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "81d4bc41d1984bca82ffec9ac1296056"
format_stamp: "Formatted at 2026-08-11 11:54:46 on dist-test-slave-bndk"
server_key: "875d49bb0ecf92fbb721344eb31d44b4"
server_key_iv: "4534fa8bb2b7e6ac34b8dc8c53ea54a3"
server_key_version: "encryptionkey@0"
I20260811 11:54:46.594348 5553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:54:46.597196 5580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:46.598904 5553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:54:46.599045 5553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "81d4bc41d1984bca82ffec9ac1296056"
format_stamp: "Formatted at 2026-08-11 11:54:46 on dist-test-slave-bndk"
server_key: "875d49bb0ecf92fbb721344eb31d44b4"
server_key_iv: "4534fa8bb2b7e6ac34b8dc8c53ea54a3"
server_key_version: "encryptionkey@0"
I20260811 11:54:46.599186 5553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:46.614521 5553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:46.618935 5553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:46.619163 5553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:46.627170 5553 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40021
I20260811 11:54:46.627190 5661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40021 every 8 connection(s)
I20260811 11:54:46.628234 5553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:54:46.631374 5662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:46.637471 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 5553
I20260811 11:54:46.637614 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:54:46.637781 5662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056: Bootstrap starting.
I20260811 11:54:46.637893 3957 external_mini_cluster.cc:1428] Setting key ad77639124e5b8d19d0b1e6499376e9e
I20260811 11:54:46.640765 5662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:46.641813 5662 log.cc:824] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:46.644346 5662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056: No bootstrap required, opened a new log
I20260811 11:54:46.648779 5662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d4bc41d1984bca82ffec9ac1296056" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40021 } }
I20260811 11:54:46.649065 5662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:46.649127 5662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d4bc41d1984bca82ffec9ac1296056, State: Initialized, Role: FOLLOWER
Aug 11 11:54:46 dist-test-slave-bndk krb5kdc[5531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 11:54:46.649721 5662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d4bc41d1984bca82ffec9ac1296056" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40021 } }
I20260811 11:54:46.649881 5662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:54:46.649951 5662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:54:46.650058 5662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:46.651357 5662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d4bc41d1984bca82ffec9ac1296056" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40021 } }
I20260811 11:54:46.651734 5662 leader_election.cc:302] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81d4bc41d1984bca82ffec9ac1296056; no voters:
I20260811 11:54:46.652077 5662 leader_election.cc:288] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:54:46.652170 5669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:46.652355 5669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056 [term 1 LEADER]: Becoming Leader. State: Replica: 81d4bc41d1984bca82ffec9ac1296056, State: Running, Role: LEADER
I20260811 11:54:46.652729 5669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d4bc41d1984bca82ffec9ac1296056" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40021 } }
I20260811 11:54:46.653522 5662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:54:46.654575 5671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81d4bc41d1984bca82ffec9ac1296056. Latest consensus state: current_term: 1 leader_uuid: "81d4bc41d1984bca82ffec9ac1296056" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d4bc41d1984bca82ffec9ac1296056" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40021 } } }
I20260811 11:54:46.654728 5671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:46.655053 5670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81d4bc41d1984bca82ffec9ac1296056" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d4bc41d1984bca82ffec9ac1296056" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40021 } } }
I20260811 11:54:46.655161 5670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:46.655176 5684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:54:46.656708 5667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:46.640426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:59889 (local address 127.0.0.1:40021)
0811 11:54:46.641305 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:46.641327 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:46.641377 (+ 50us) server_negotiation.cc:407] Connection header received
0811 11:54:46.642338 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:46.642364 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:46.642743 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:46.643180 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:46.643981 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:46.645494 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:46.646435 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:46.647169 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:46.647362 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:46.650321 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:46.650328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:46.650346 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:46.650385 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:46.652418 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:46.653104 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:46.653112 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:46.653118 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:46.653290 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:46.653653 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:46.653656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:46.653658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:46.654186 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:46.654435 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:46.655439 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0811 11:54:46.655688 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":279,"threads_started":1}
I20260811 11:54:46.658584 5684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:54:46.660161 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:54:46.665575 5684 catalog_manager.cc:1347] Generated new cluster ID: 9e662aa56ccd4e8d98aa41ac34b2df18
I20260811 11:54:46.665797 5684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:54:46.672715 5667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:46.662385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:40561 (local address 127.0.0.1:40021)
0811 11:54:46.662488 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:46.662492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:46.662705 (+ 213us) server_negotiation.cc:407] Connection header received
0811 11:54:46.662838 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:46.662841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:46.662896 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:46.662987 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:46.663782 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:46.665011 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:46.666216 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:46.666586 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:46.666731 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:46.668394 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:46.668398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:46.668400 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:46.668428 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:46.670204 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:46.670780 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:46.670783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:46.670785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:46.670880 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:46.671237 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:46.671241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:46.671242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:46.671522 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:46.671712 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:46.672076 (+ 364us) server_negotiation.cc:299] Negotiation successful
0811 11:54:46.672187 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":28}
I20260811 11:54:46.673070 5684 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:54:46.674285 5684 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:54:46.690816 5684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81d4bc41d1984bca82ffec9ac1296056: Generated new TSK 0
I20260811 11:54:46.691879 5684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:54:46.700976 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 5553
W20260811 11:54:46.724922 5528 connection.cc:535] client connection to 127.0.0.1:40021 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40021): Connection reset by peer (error 104)
2026-08-11T11:54:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1609 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:54:46 dist-test-slave-bndk krb5kdc[5714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:54:46 dist-test-slave-bndk krb5kdc[5714](info): set up 2 sockets
Aug 11 11:54:46 dist-test-slave-bndk krb5kdc[5714](info): commencing operation
krb5kdc: starting...
W20260811 11:54:47.915426 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:54:48 dist-test-slave-bndk krb5kdc[5714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:54:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:54:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:54:48.061319 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44897
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:48.184811 5733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:48.185128 5733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:48.185191 5733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:48.189479 5733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:54:48.189564 5733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:48.189594 5733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:48.189615 5733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:54:48.189637 5733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:54:48.195652 5733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:48.196933 5733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:48.198055 5733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:48.207255 5747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:48.207255 5744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:48.207312 5742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:48.207299 5733 server_base.cc:1034] running on GCE node
I20260811 11:54:48.208310 5733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:48.209597 5733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:48.210798 5733 hybrid_clock.cc:648] HybridClock initialized: now 1786449288210758 us; error 57 us; skew 500 ppm
Aug 11 11:54:48 dist-test-slave-bndk krb5kdc[5714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:48.213842 5733 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:54:48.214874 5733 webserver.cc:460] Webserver started at http://127.0.0.1:35869/ using document root <none> and password file <none>
I20260811 11:54:48.215445 5733 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:48.215508 5733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:48.215709 5733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:48.217952 5733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "730083b3913744249067f3922953a2ff"
format_stamp: "Formatted at 2026-08-11 11:54:48 on dist-test-slave-bndk"
server_key: "839acf258d94f2d7412aa20240d667e2"
server_key_iv: "ca2acbc7d17a6cd5aa90da95cec48369"
server_key_version: "encryptionkey@0"
I20260811 11:54:48.218518 5733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "730083b3913744249067f3922953a2ff"
format_stamp: "Formatted at 2026-08-11 11:54:48 on dist-test-slave-bndk"
server_key: "839acf258d94f2d7412aa20240d667e2"
server_key_iv: "ca2acbc7d17a6cd5aa90da95cec48369"
server_key_version: "encryptionkey@0"
I20260811 11:54:48.222503 5733 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260811 11:54:48.225267 5762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:48.226588 5733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 11:54:48.226725 5733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "730083b3913744249067f3922953a2ff"
format_stamp: "Formatted at 2026-08-11 11:54:48 on dist-test-slave-bndk"
server_key: "839acf258d94f2d7412aa20240d667e2"
server_key_iv: "ca2acbc7d17a6cd5aa90da95cec48369"
server_key_version: "encryptionkey@0"
I20260811 11:54:48.226851 5733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:48.245572 5733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:48.249063 5733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:48.249301 5733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:48.256630 5733 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44897
I20260811 11:54:48.256651 5838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44897 every 8 connection(s)
I20260811 11:54:48.257781 5733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:54:48.260120 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 5733
I20260811 11:54:48.260281 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:54:48.260592 3957 external_mini_cluster.cc:1428] Setting key a9b0e50fa7bed8fd6b0088286afc4dc8
I20260811 11:54:48.261191 5839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:48.267910 5839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff: Bootstrap starting.
I20260811 11:54:48.270006 5842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:48.262182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:56831 (local address 127.0.0.1:44897)
0811 11:54:48.262692 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:48.262709 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:48.262747 (+ 38us) server_negotiation.cc:407] Connection header received
0811 11:54:48.263456 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:48.263475 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:48.263849 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:48.264178 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:48.264599 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:48.265683 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:48.266567 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:48.267061 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:48.267210 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:48.268164 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:48.268170 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:48.268183 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:48.268594 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:48.268662 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:48.268686 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:48.268804 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:48.268989 (+ 185us) server_negotiation.cc:299] Negotiation successful
0811 11:54:48.269130 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":171,"threads_started":1}
I20260811 11:54:48.270471 5839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:48.271380 5839 log.cc:824] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:48.273533 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:54:48.273834 5839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff: No bootstrap required, opened a new log
I20260811 11:54:48.278278 5839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730083b3913744249067f3922953a2ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44897 } }
I20260811 11:54:48.278609 5839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:48.278678 5839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730083b3913744249067f3922953a2ff, State: Initialized, Role: FOLLOWER
I20260811 11:54:48.279213 5839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730083b3913744249067f3922953a2ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44897 } }
I20260811 11:54:48.279412 5839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:54:48.279538 5839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:54:48.279666 5839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:48.279847 5842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:48.275526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:59461 (local address 127.0.0.1:44897)
0811 11:54:48.275655 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:48.275660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:48.275861 (+ 201us) server_negotiation.cc:407] Connection header received
0811 11:54:48.276001 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:48.276004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:48.276058 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:48.276118 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:48.276456 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:48.276985 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:48.278126 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:48.278381 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:48.278469 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:48.279341 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:48.279344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:48.279346 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:48.279359 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:48.279385 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:48.279396 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:48.279478 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:48.279654 (+ 176us) server_negotiation.cc:299] Negotiation successful
0811 11:54:48.279699 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 11:54:48.280972 5839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730083b3913744249067f3922953a2ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44897 } }
I20260811 11:54:48.281388 5839 leader_election.cc:302] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 730083b3913744249067f3922953a2ff; no voters:
I20260811 11:54:48.281745 5839 leader_election.cc:288] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:54:48.281859 5851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:48.282086 5851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff [term 1 LEADER]: Becoming Leader. State: Replica: 730083b3913744249067f3922953a2ff, State: Running, Role: LEADER
I20260811 11:54:48.282469 5851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730083b3913744249067f3922953a2ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44897 } }
I20260811 11:54:48.283041 5839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:54:48.284147 5853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "730083b3913744249067f3922953a2ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730083b3913744249067f3922953a2ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44897 } } }
I20260811 11:54:48.284147 5854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 730083b3913744249067f3922953a2ff. Latest consensus state: current_term: 1 leader_uuid: "730083b3913744249067f3922953a2ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730083b3913744249067f3922953a2ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44897 } } }
I20260811 11:54:48.284415 5854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:48.284415 5853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:48.284932 5868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:54:48.287894 5868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:54:48.294201 5868 catalog_manager.cc:1347] Generated new cluster ID: 9c427e9510454b92af5df5d49c4fecda
I20260811 11:54:48.294318 5868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:54:48.310585 5868 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:54:48.312044 5868 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:54:48.328680 5868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 730083b3913744249067f3922953a2ff: Generated new TSK 0
I20260811 11:54:48.329514 5868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:54:48.342284 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 5733
W20260811 11:54:48.354724 5709 connection.cc:535] client connection to 127.0.0.1:44897 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44897): Connection reset by peer (error 104)
2026-08-11T11:54:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1629 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:54:48 dist-test-slave-bndk krb5kdc[5892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:54:48 dist-test-slave-bndk krb5kdc[5892](info): set up 2 sockets
Aug 11 11:54:48 dist-test-slave-bndk krb5kdc[5892](info): commencing operation
krb5kdc: starting...
W20260811 11:54:49.526118 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:54:49 dist-test-slave-bndk krb5kdc[5892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:54:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:54:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:54:49.677769 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46523
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:49.803076 5919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:49.803476 5919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:49.803563 5919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:49.808642 5919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:54:49.808730 5919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:49.808753 5919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:49.808769 5919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:54:49.808784 5919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:54:49.814507 5919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:49.815635 5919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:49.816696 5919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:49.825656 5930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:49.825680 5924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:49.825835 5919 server_base.cc:1034] running on GCE node
W20260811 11:54:49.825680 5926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:49.826627 5919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:49.827831 5919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:49.829021 5919 hybrid_clock.cc:648] HybridClock initialized: now 1786449289829009 us; error 28 us; skew 500 ppm
Aug 11 11:54:49 dist-test-slave-bndk krb5kdc[5892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:49.832015 5919 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:54:49.832996 5919 webserver.cc:460] Webserver started at http://127.0.0.1:36045/ using document root <none> and password file <none>
I20260811 11:54:49.833621 5919 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:49.833698 5919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:49.833928 5919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:49.836114 5919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "9cba810e31324290b6608c5cb24c9759"
format_stamp: "Formatted at 2026-08-11 11:54:49 on dist-test-slave-bndk"
server_key: "19a9e03b9d34fe64a261ef3284fd8dfb"
server_key_iv: "2e2a7c5f750b43f16f38d62bb3038c29"
server_key_version: "encryptionkey@0"
I20260811 11:54:49.836714 5919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "9cba810e31324290b6608c5cb24c9759"
format_stamp: "Formatted at 2026-08-11 11:54:49 on dist-test-slave-bndk"
server_key: "19a9e03b9d34fe64a261ef3284fd8dfb"
server_key_iv: "2e2a7c5f750b43f16f38d62bb3038c29"
server_key_version: "encryptionkey@0"
I20260811 11:54:49.840718 5919 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 11:54:49.843533 5946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:49.844828 5919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 11:54:49.844960 5919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "9cba810e31324290b6608c5cb24c9759"
format_stamp: "Formatted at 2026-08-11 11:54:49 on dist-test-slave-bndk"
server_key: "19a9e03b9d34fe64a261ef3284fd8dfb"
server_key_iv: "2e2a7c5f750b43f16f38d62bb3038c29"
server_key_version: "encryptionkey@0"
I20260811 11:54:49.845079 5919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:49.849884 5919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:49.853531 5919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:49.854118 5919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:49.863018 5919 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46523
I20260811 11:54:49.863046 6030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46523 every 8 connection(s)
I20260811 11:54:49.864033 5919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:54:49.867122 6031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:49.867296 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 5919
I20260811 11:54:49.867456 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:54:49.867770 3957 external_mini_cluster.cc:1428] Setting key 3383ca11b71ed44e884bc518aed7a7d1
I20260811 11:54:49.873759 6034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:49.869336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:41709 (local address 127.0.0.1:46523)
0811 11:54:49.869903 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:49.869924 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:49.869963 (+ 39us) server_negotiation.cc:407] Connection header received
0811 11:54:49.870808 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:49.870841 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:49.871159 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:49.871657 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:49.872486 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:49.872491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:49.872504 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:49.872514 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:49.872572 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:49.872592 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:49.872726 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:49.872805 (+ 79us) server_negotiation.cc:299] Negotiation successful
0811 11:54:49.872968 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":166,"threads_started":1}
I20260811 11:54:49.874732 6031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759: Bootstrap starting.
I20260811 11:54:49.877429 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:54:49.877586 6031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:49.878396 6031 log.cc:824] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:49.881069 6031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759: No bootstrap required, opened a new log
I20260811 11:54:49.881687 6034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:49.879973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:50223 (local address 127.0.0.1:46523)
0811 11:54:49.880095 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:49.880101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:49.880306 (+ 205us) server_negotiation.cc:407] Connection header received
0811 11:54:49.880438 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:49.880441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:49.880475 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:49.880538 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:49.881312 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:49.881315 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:49.881318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:49.881331 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:49.881362 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:49.881376 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:49.881469 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:49.881520 (+ 51us) server_negotiation.cc:299] Negotiation successful
0811 11:54:49.881566 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 11:54:49.885298 6031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cba810e31324290b6608c5cb24c9759" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } }
I20260811 11:54:49.885572 6031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:49.885624 6031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cba810e31324290b6608c5cb24c9759, State: Initialized, Role: FOLLOWER
I20260811 11:54:49.886111 6031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cba810e31324290b6608c5cb24c9759" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } }
I20260811 11:54:49.886253 6031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:54:49.886330 6031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:54:49.886421 6031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:49.887677 6031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cba810e31324290b6608c5cb24c9759" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } }
I20260811 11:54:49.888051 6031 leader_election.cc:302] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cba810e31324290b6608c5cb24c9759; no voters:
I20260811 11:54:49.888434 6031 leader_election.cc:288] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:54:49.888602 6044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:49.888844 6044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759 [term 1 LEADER]: Becoming Leader. State: Replica: 9cba810e31324290b6608c5cb24c9759, State: Running, Role: LEADER
I20260811 11:54:49.889195 6044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cba810e31324290b6608c5cb24c9759" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } }
I20260811 11:54:49.889804 6031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:54:49.890954 6045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cba810e31324290b6608c5cb24c9759" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cba810e31324290b6608c5cb24c9759" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } } }
I20260811 11:54:49.891103 6045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:49.891477 6047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cba810e31324290b6608c5cb24c9759. Latest consensus state: current_term: 1 leader_uuid: "9cba810e31324290b6608c5cb24c9759" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cba810e31324290b6608c5cb24c9759" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } } }
I20260811 11:54:49.891597 6054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:54:49.891659 6047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:49.895895 6054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:54:49.902158 6054 catalog_manager.cc:1347] Generated new cluster ID: f4b404457b674cbeb9652fd2304253f2
I20260811 11:54:49.902261 6054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:54:49.913897 6054 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:54:49.914259 6054 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:54:49.919941 6054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cba810e31324290b6608c5cb24c9759: Generated new TSK 0
I20260811 11:54:49.920801 6054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:54:49.931990 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 5919
2026-08-11T11:54:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1597 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:54:50 dist-test-slave-bndk krb5kdc[6071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:54:50 dist-test-slave-bndk krb5kdc[6071](info): set up 2 sockets
Aug 11 11:54:50 dist-test-slave-bndk krb5kdc[6071](info): commencing operation
krb5kdc: starting...
W20260811 11:54:51.121299 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:54:51 dist-test-slave-bndk krb5kdc[6071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:54:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:54:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:54:51.257781 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36279
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:51.378983 6099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:51.379655 6099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:51.379773 6099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:51.384032 6099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:54:51.384119 6099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:51.384146 6099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:51.384168 6099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:54:51.384188 6099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:54:51.390020 6099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36279
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:51.391552 6099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:51.392705 6099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:51.402196 6107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:51.402191 6109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:51.402191 6105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:51.402508 6099 server_base.cc:1034] running on GCE node
I20260811 11:54:51.403123 6099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:51.404347 6099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:51.405663 6099 hybrid_clock.cc:648] HybridClock initialized: now 1786449291405618 us; error 38 us; skew 500 ppm
Aug 11 11:54:51 dist-test-slave-bndk krb5kdc[6071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:51.409421 6099 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:54:51.410450 6099 webserver.cc:460] Webserver started at http://127.0.0.1:44887/ using document root <none> and password file <none>
I20260811 11:54:51.411018 6099 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:51.411078 6099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:51.411274 6099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:51.413504 6099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "c1bcde23cda4443a9f09b9fb650de543"
format_stamp: "Formatted at 2026-08-11 11:54:51 on dist-test-slave-bndk"
server_key: "7c55a3942b211dcc083d21ea2becc60d"
server_key_iv: "14c2234fa2d1cc3d8ac92884114ee3a4"
server_key_version: "encryptionkey@0"
I20260811 11:54:51.414059 6099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "c1bcde23cda4443a9f09b9fb650de543"
format_stamp: "Formatted at 2026-08-11 11:54:51 on dist-test-slave-bndk"
server_key: "7c55a3942b211dcc083d21ea2becc60d"
server_key_iv: "14c2234fa2d1cc3d8ac92884114ee3a4"
server_key_version: "encryptionkey@0"
I20260811 11:54:51.417906 6099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 11:54:51.420804 6125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:51.422228 6099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 11:54:51.422355 6099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "c1bcde23cda4443a9f09b9fb650de543"
format_stamp: "Formatted at 2026-08-11 11:54:51 on dist-test-slave-bndk"
server_key: "7c55a3942b211dcc083d21ea2becc60d"
server_key_iv: "14c2234fa2d1cc3d8ac92884114ee3a4"
server_key_version: "encryptionkey@0"
I20260811 11:54:51.422477 6099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:51.459923 6099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:51.463543 6099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:51.463721 6099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:51.471326 6099 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36279
I20260811 11:54:51.471424 6218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36279 every 8 connection(s)
I20260811 11:54:51.472362 6099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:54:51.475411 6219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:51.477110 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 6099
I20260811 11:54:51.477269 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:54:51.477519 3957 external_mini_cluster.cc:1428] Setting key 567f89be010b37e622170bc001c6ec27
I20260811 11:54:51.481941 6219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543: Bootstrap starting.
I20260811 11:54:51.484750 6219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:51.485576 6219 log.cc:824] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543: Log is configured to *not* fsync() on all Append() calls
Aug 11 11:54:51 dist-test-slave-bndk krb5kdc[6071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 11:54:51.487882 6219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543: No bootstrap required, opened a new log
I20260811 11:54:51.492668 6219 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bcde23cda4443a9f09b9fb650de543" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36279 } }
I20260811 11:54:51.492964 6219 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:51.493028 6219 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1bcde23cda4443a9f09b9fb650de543, State: Initialized, Role: FOLLOWER
I20260811 11:54:51.493108 6224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:51.478993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:49581 (local address 127.0.0.1:36279)
0811 11:54:51.479524 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:51.479541 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:51.479581 (+ 40us) server_negotiation.cc:407] Connection header received
0811 11:54:51.480277 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:51.480296 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:51.480676 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:51.481012 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:51.481871 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:51.482996 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:51.483877 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:51.484366 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:51.484541 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:51.487262 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:51.487269 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:51.487288 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:51.487321 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:51.489259 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:51.489862 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:51.489868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:51.489875 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:51.490035 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:51.490348 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:51.490352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:51.490354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:51.490938 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:51.491155 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:51.491757 (+ 602us) server_negotiation.cc:299] Negotiation successful
0811 11:54:51.492037 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":170,"threads_started":1}
I20260811 11:54:51.493556 6219 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bcde23cda4443a9f09b9fb650de543" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36279 } }
I20260811 11:54:51.493695 6219 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:54:51.493755 6219 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:54:51.493851 6219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:51.494830 6219 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bcde23cda4443a9f09b9fb650de543" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36279 } }
I20260811 11:54:51.495169 6219 leader_election.cc:302] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1bcde23cda4443a9f09b9fb650de543; no voters:
I20260811 11:54:51.495553 6219 leader_election.cc:288] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:54:51.495671 6229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:51.495919 6229 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543 [term 1 LEADER]: Becoming Leader. State: Replica: c1bcde23cda4443a9f09b9fb650de543, State: Running, Role: LEADER
I20260811 11:54:51.496353 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:54:51.496331 6229 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bcde23cda4443a9f09b9fb650de543" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36279 } }
I20260811 11:54:51.496955 6219 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:54:51.498056 6231 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1bcde23cda4443a9f09b9fb650de543. Latest consensus state: current_term: 1 leader_uuid: "c1bcde23cda4443a9f09b9fb650de543" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bcde23cda4443a9f09b9fb650de543" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36279 } } }
I20260811 11:54:51.498070 6233 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1bcde23cda4443a9f09b9fb650de543" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bcde23cda4443a9f09b9fb650de543" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36279 } } }
I20260811 11:54:51.498226 6231 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:51.498226 6233 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:51.500087 6244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:54:51.504491 6244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:54:51.508972 6224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:51.499398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:47059 (local address 127.0.0.1:36279)
0811 11:54:51.499578 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:51.499584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:51.500362 (+ 778us) server_negotiation.cc:407] Connection header received
0811 11:54:51.500539 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:51.500542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:51.500601 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:51.500695 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:51.501594 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:51.502115 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:51.503311 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:51.503763 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:51.503938 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:51.505781 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:51.505785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:51.505787 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:51.505816 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:51.507173 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:51.507689 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:51.507692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:51.507693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:51.507776 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:51.508089 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:51.508091 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:51.508092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:51.508293 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:51.508443 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:51.508708 (+ 265us) server_negotiation.cc:299] Negotiation successful
0811 11:54:51.508810 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 11:54:51.512439 6244 catalog_manager.cc:1347] Generated new cluster ID: dc0a7dd315fb45aeb9137cdd526febf8
I20260811 11:54:51.512518 6244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:54:51.532631 6244 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:54:51.534108 6244 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:54:51.549243 6244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1bcde23cda4443a9f09b9fb650de543: Generated new TSK 0
I20260811 11:54:51.550190 6244 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:54:51.555830 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 6099
W20260811 11:54:51.570223 6066 connection.cc:535] client connection to 127.0.0.1:36279 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36279): Connection reset by peer (error 104)
2026-08-11T11:54:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1618 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:54:51 dist-test-slave-bndk krb5kdc[6266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:54:51 dist-test-slave-bndk krb5kdc[6266](info): set up 2 sockets
Aug 11 11:54:51 dist-test-slave-bndk krb5kdc[6266](info): commencing operation
krb5kdc: starting...
W20260811 11:54:52.738032 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:54:52 dist-test-slave-bndk krb5kdc[6266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:54:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:54:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:54:52.876915 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44395
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:52.998201 6285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:52.998553 6285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:52.998615 6285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:53.002861 6285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:54:53.002949 6285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:53.002976 6285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:53.002998 6285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:54:53.003019 6285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:54:53.009143 6285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44395
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:53.010435 6285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:53.011543 6285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:53.020588 6295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:53.020586 6296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:53.020632 6285 server_base.cc:1034] running on GCE node
W20260811 11:54:53.020576 6298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:53.021526 6285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:53.022735 6285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:53.023926 6285 hybrid_clock.cc:648] HybridClock initialized: now 1786449293023909 us; error 31 us; skew 500 ppm
Aug 11 11:54:53 dist-test-slave-bndk krb5kdc[6266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:53.027086 6285 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:54:53.028113 6285 webserver.cc:460] Webserver started at http://127.0.0.1:46277/ using document root <none> and password file <none>
I20260811 11:54:53.028729 6285 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:53.028805 6285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:53.029042 6285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:53.031294 6285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "0e7ab3f033544f209ff8b478edc7cc63"
format_stamp: "Formatted at 2026-08-11 11:54:53 on dist-test-slave-bndk"
server_key: "de9bf69091ca0b8e755cb10b4963c102"
server_key_iv: "e42b1134729475a0706e0727a929a1a5"
server_key_version: "encryptionkey@0"
I20260811 11:54:53.031939 6285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "0e7ab3f033544f209ff8b478edc7cc63"
format_stamp: "Formatted at 2026-08-11 11:54:53 on dist-test-slave-bndk"
server_key: "de9bf69091ca0b8e755cb10b4963c102"
server_key_iv: "e42b1134729475a0706e0727a929a1a5"
server_key_version: "encryptionkey@0"
I20260811 11:54:53.036203 6285 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 11:54:53.039076 6321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:53.040529 6285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 11:54:53.040674 6285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "0e7ab3f033544f209ff8b478edc7cc63"
format_stamp: "Formatted at 2026-08-11 11:54:53 on dist-test-slave-bndk"
server_key: "de9bf69091ca0b8e755cb10b4963c102"
server_key_iv: "e42b1134729475a0706e0727a929a1a5"
server_key_version: "encryptionkey@0"
I20260811 11:54:53.040809 6285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:53.060246 6285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:53.064224 6285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:53.064455 6285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:53.073877 6285 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44395
I20260811 11:54:53.073942 6407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44395 every 8 connection(s)
I20260811 11:54:53.075620 6285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:54:53.079066 6409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:53.085286 6409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63: Bootstrap starting.
I20260811 11:54:53.085552 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 6285
I20260811 11:54:53.085681 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:54:53.085954 3957 external_mini_cluster.cc:1428] Setting key f4b1dcbabbe021a45f769b216349eb28
I20260811 11:54:53.088145 6409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:53.089118 6409 log.cc:824] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:53.091361 6409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63: No bootstrap required, opened a new log
I20260811 11:54:53.095641 6409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7ab3f033544f209ff8b478edc7cc63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44395 } }
I20260811 11:54:53.095956 6409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:53.096010 6409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e7ab3f033544f209ff8b478edc7cc63, State: Initialized, Role: FOLLOWER
I20260811 11:54:53.096128 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:53.087459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:40323 (local address 127.0.0.1:44395)
0811 11:54:53.087932 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:53.087952 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:53.087997 (+ 45us) server_negotiation.cc:407] Connection header received
0811 11:54:53.088936 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:53.088964 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:53.089492 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:53.089881 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:53.090313 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:53.091303 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:53.092201 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:53.092693 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:53.092858 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:53.093832 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:53.093841 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:53.093872 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:53.094441 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:53.094520 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:53.094553 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:53.094704 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:53.094910 (+ 206us) server_negotiation.cc:299] Negotiation successful
0811 11:54:53.095063 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":117,"threads_started":1}
I20260811 11:54:53.096522 6409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7ab3f033544f209ff8b478edc7cc63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44395 } }
I20260811 11:54:53.096691 6409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:54:53.096760 6409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:54:53.096853 6409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:53.098016 6409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7ab3f033544f209ff8b478edc7cc63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44395 } }
I20260811 11:54:53.098394 6409 leader_election.cc:302] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e7ab3f033544f209ff8b478edc7cc63; no voters:
I20260811 11:54:53.098739 6409 leader_election.cc:288] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:54:53.098852 6419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:53.099082 6419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63 [term 1 LEADER]: Becoming Leader. State: Replica: 0e7ab3f033544f209ff8b478edc7cc63, State: Running, Role: LEADER
I20260811 11:54:53.099383 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:54:53.099588 6419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7ab3f033544f209ff8b478edc7cc63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44395 } }
I20260811 11:54:53.100155 6409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:54:53.101559 6422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e7ab3f033544f209ff8b478edc7cc63. Latest consensus state: current_term: 1 leader_uuid: "0e7ab3f033544f209ff8b478edc7cc63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7ab3f033544f209ff8b478edc7cc63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44395 } } }
I20260811 11:54:53.101693 6422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:53.102594 6421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e7ab3f033544f209ff8b478edc7cc63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7ab3f033544f209ff8b478edc7cc63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44395 } } }
I20260811 11:54:53.102834 6421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63 [sys.catalog]: This master's current role is: LEADER
W20260811 11:54:53.105819 6447 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:54:53.105901 6447 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:54:53.105997 6440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:54:53.106990 6440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:54:53.107129 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:53.102051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:47259 (local address 127.0.0.1:44395)
0811 11:54:53.102159 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:53.102163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:53.102742 (+ 579us) server_negotiation.cc:407] Connection header received
0811 11:54:53.102881 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:53.102884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:53.102938 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:53.103018 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:53.103321 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:53.103845 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:53.104974 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:53.105339 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:53.105455 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:53.106435 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:53.106439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:53.106442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:53.106457 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:53.106486 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:53.106501 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:53.106684 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:53.106876 (+ 192us) server_negotiation.cc:299] Negotiation successful
0811 11:54:53.106919 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 11:54:53.112857 6440 catalog_manager.cc:1347] Generated new cluster ID: 604a772ed72243b0a7330c2066d58412
I20260811 11:54:53.112949 6440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:54:53.143344 6440 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:54:53.144677 6440 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:54:53.161562 6440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e7ab3f033544f209ff8b478edc7cc63: Generated new TSK 0
I20260811 11:54:53.162418 6440 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:54:53.175904 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 6285
W20260811 11:54:53.198222 6262 connection.cc:535] client connection to 127.0.0.1:44395 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44395): Connection reset by peer (error 104)
2026-08-11T11:54:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1627 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:54:53 dist-test-slave-bndk krb5kdc[6454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:54:53 dist-test-slave-bndk krb5kdc[6454](info): set up 2 sockets
Aug 11 11:54:53 dist-test-slave-bndk krb5kdc[6454](info): commencing operation
krb5kdc: starting...
W20260811 11:54:54.374675 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:54:54 dist-test-slave-bndk krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:54:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:54:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 11:54:54.516113 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45175
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:54.648420 6477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:54.648742 6477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:54.648788 6477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:54.653087 6477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:54:54.653173 6477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:54.653481 6477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:54.653558 6477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:54:54.653597 6477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:54:54.660979 6477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45175
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:54.662257 6477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:54.663364 6477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:54.672618 6487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:54.672627 6483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:54.672695 6485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:54.673008 6477 server_base.cc:1034] running on GCE node
I20260811 11:54:54.673673 6477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:54.674961 6477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:54.676156 6477 hybrid_clock.cc:648] HybridClock initialized: now 1786449294676130 us; error 42 us; skew 500 ppm
Aug 11 11:54:54 dist-test-slave-bndk krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:54.682739 6477 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 11:54:54.683795 6477 webserver.cc:460] Webserver started at http://127.0.0.1:38415/ using document root <none> and password file <none>
I20260811 11:54:54.684398 6477 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:54.684471 6477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:54.684687 6477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:54.687103 6477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "6d47975166bc45c7a922a9f3b3c4b903"
format_stamp: "Formatted at 2026-08-11 11:54:54 on dist-test-slave-bndk"
server_key: "7542692c748d82797d9a4867799d8cfa"
server_key_iv: "5d53c29c5bd86245260197898908a1e6"
server_key_version: "encryptionkey@0"
I20260811 11:54:54.687819 6477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "6d47975166bc45c7a922a9f3b3c4b903"
format_stamp: "Formatted at 2026-08-11 11:54:54 on dist-test-slave-bndk"
server_key: "7542692c748d82797d9a4867799d8cfa"
server_key_iv: "5d53c29c5bd86245260197898908a1e6"
server_key_version: "encryptionkey@0"
I20260811 11:54:54.692518 6477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260811 11:54:54.695292 6498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:54.696842 6477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260811 11:54:54.696991 6477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "6d47975166bc45c7a922a9f3b3c4b903"
format_stamp: "Formatted at 2026-08-11 11:54:54 on dist-test-slave-bndk"
server_key: "7542692c748d82797d9a4867799d8cfa"
server_key_iv: "5d53c29c5bd86245260197898908a1e6"
server_key_version: "encryptionkey@0"
I20260811 11:54:54.697110 6477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:54.702453 6477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:54.705773 6477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:54.705958 6477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:54.713485 6477 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45175
I20260811 11:54:54.713641 6582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45175 every 8 connection(s)
I20260811 11:54:54.714569 6477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:54:54.714861 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 6477
I20260811 11:54:54.714983 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:54:54.715301 3957 external_mini_cluster.cc:1428] Setting key 5f6843065ea7a85357b0624d53b7a6d0
I20260811 11:54:54.717851 6583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:54.720819 6587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:54.716791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:45739 (local address 127.0.0.1:45175)
0811 11:54:54.717253 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:54.717274 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:54.717321 (+ 47us) server_negotiation.cc:407] Connection header received
0811 11:54:54.718061 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:54.718080 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:54.718379 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:54.718681 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:54.719553 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:54.719558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:54.719571 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:54.719583 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:54.719644 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:54.719665 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:54.719803 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:54.719882 (+ 79us) server_negotiation.cc:299] Negotiation successful
0811 11:54:54.720047 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":117,"threads_started":1}
I20260811 11:54:54.723690 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:54:54.724277 6583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903: Bootstrap starting.
I20260811 11:54:54.727213 6583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:54.727909 6587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:54.726155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:46143 (local address 127.0.0.1:45175)
0811 11:54:54.726296 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:54.726300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:54.726494 (+ 194us) server_negotiation.cc:407] Connection header received
0811 11:54:54.726608 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:54.726611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:54.726646 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:54.726716 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:54.727477 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:54.727480 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:54.727482 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:54.727493 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:54.727523 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:54.727534 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:54.727608 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:54.727741 (+ 133us) server_negotiation.cc:299] Negotiation successful
0811 11:54:54.727783 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 11:54:54.728210 6583 log.cc:824] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:54.730319 6583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903: No bootstrap required, opened a new log
I20260811 11:54:54.734163 6583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d47975166bc45c7a922a9f3b3c4b903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45175 } }
I20260811 11:54:54.734424 6583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:54.734467 6583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d47975166bc45c7a922a9f3b3c4b903, State: Initialized, Role: FOLLOWER
I20260811 11:54:54.734901 6583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d47975166bc45c7a922a9f3b3c4b903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45175 } }
I20260811 11:54:54.735039 6583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:54:54.735098 6583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:54:54.735194 6583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:54.736181 6583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d47975166bc45c7a922a9f3b3c4b903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45175 } }
I20260811 11:54:54.736629 6583 leader_election.cc:302] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d47975166bc45c7a922a9f3b3c4b903; no voters:
I20260811 11:54:54.737007 6583 leader_election.cc:288] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:54:54.737155 6598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:54.737490 6598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903 [term 1 LEADER]: Becoming Leader. State: Replica: 6d47975166bc45c7a922a9f3b3c4b903, State: Running, Role: LEADER
I20260811 11:54:54.737810 6598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d47975166bc45c7a922a9f3b3c4b903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45175 } }
I20260811 11:54:54.738294 6583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:54:54.739464 6600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d47975166bc45c7a922a9f3b3c4b903. Latest consensus state: current_term: 1 leader_uuid: "6d47975166bc45c7a922a9f3b3c4b903" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d47975166bc45c7a922a9f3b3c4b903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45175 } } }
I20260811 11:54:54.739588 6600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:54.739951 6599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d47975166bc45c7a922a9f3b3c4b903" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d47975166bc45c7a922a9f3b3c4b903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45175 } } }
I20260811 11:54:54.740008 6611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:54:54.740082 6599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:54.743192 6611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:54:54.749625 6611 catalog_manager.cc:1347] Generated new cluster ID: b5b5e85b37b14b82806b7ee455887e53
I20260811 11:54:54.749702 6611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:54:54.770283 6611 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:54:54.770772 6611 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:54:54.780269 6611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d47975166bc45c7a922a9f3b3c4b903: Generated new TSK 0
I20260811 11:54:54.781122 6611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:54:54.791281 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 6477
2026-08-11T11:54:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1617 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:54:54 dist-test-slave-bndk krb5kdc[6632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:54:54 dist-test-slave-bndk krb5kdc[6632](info): set up 2 sockets
Aug 11 11:54:54 dist-test-slave-bndk krb5kdc[6632](info): commencing operation
krb5kdc: starting...
W20260811 11:54:56.007684 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:54:56 dist-test-slave-bndk krb5kdc[6632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:54:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:54:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:54:56.148914 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:37243
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:44347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:37243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:56.270431 6647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:56.270794 6647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:56.270864 6647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:56.275131 6647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:54:56.275213 6647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:56.275241 6647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:56.275264 6647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:54:56.275285 6647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:54:56.280994 6647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:44347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:37243
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:37243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:56.282245 6647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:56.283296 6647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:56.292335 6655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:56.292315 6656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:56.292316 6661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:56.292673 6647 server_base.cc:1034] running on GCE node
I20260811 11:54:56.293761 6647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:56.295415 6647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:56.296695 6647 hybrid_clock.cc:648] HybridClock initialized: now 1786449296296649 us; error 49 us; skew 500 ppm
Aug 11 11:54:56 dist-test-slave-bndk krb5kdc[6632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449296, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:56.300264 6647 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:54:56.301347 6647 webserver.cc:460] Webserver started at http://127.3.221.126:43073/ using document root <none> and password file <none>
I20260811 11:54:56.301952 6647 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:56.302028 6647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:56.302254 6647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:56.304478 6647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "3758e78be88f4e10b3ab6f0f17622a94"
format_stamp: "Formatted at 2026-08-11 11:54:56 on dist-test-slave-bndk"
server_key: "0530f3d25e452a63e1c3fc87281af55c"
server_key_iv: "5f95fd02ab39e435145a4b7c931b5554"
server_key_version: "encryptionkey@0"
I20260811 11:54:56.305058 6647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "3758e78be88f4e10b3ab6f0f17622a94"
format_stamp: "Formatted at 2026-08-11 11:54:56 on dist-test-slave-bndk"
server_key: "0530f3d25e452a63e1c3fc87281af55c"
server_key_iv: "5f95fd02ab39e435145a4b7c931b5554"
server_key_version: "encryptionkey@0"
I20260811 11:54:56.309084 6647 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 11:54:56.312121 6673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:56.313781 6647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:54:56.313941 6647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "3758e78be88f4e10b3ab6f0f17622a94"
format_stamp: "Formatted at 2026-08-11 11:54:56 on dist-test-slave-bndk"
server_key: "0530f3d25e452a63e1c3fc87281af55c"
server_key_iv: "5f95fd02ab39e435145a4b7c931b5554"
server_key_version: "encryptionkey@0"
I20260811 11:54:56.314054 6647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:56.331862 6647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:56.335474 6647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:56.335690 6647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:56.343750 6647 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:37243
I20260811 11:54:56.343822 6764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:37243 every 8 connection(s)
I20260811 11:54:56.344823 6647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:54:56.347869 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 6647
I20260811 11:54:56.348026 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:54:56.348135 6765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:56.348351 3957 external_mini_cluster.cc:1428] Setting key 2f1ad9f8746f0049cbe9d6ad0230df76
I20260811 11:54:56.356451 6765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94: Bootstrap starting.
Aug 11 11:54:56 dist-test-slave-bndk krb5kdc[6632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:54:56.359376 6765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:56.360244 6765 log.cc:824] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:56.362803 6765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94: No bootstrap required, opened a new log
I20260811 11:54:56.365396 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:56.349793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42229 (local address 127.3.221.126:37243)
0811 11:54:56.350241 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:56.350260 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:56.350297 (+ 37us) server_negotiation.cc:407] Connection header received
0811 11:54:56.351021 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:56.351039 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:56.351434 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:56.351725 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:56.352660 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:56.353800 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:56.355211 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:56.355723 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:56.355856 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:56.358833 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:56.358838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:56.358853 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:56.358887 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:56.361397 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:56.362049 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:56.362054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:56.362059 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:56.362188 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:56.362558 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:56.362560 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:56.362561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:56.363035 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:56.363203 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:56.363868 (+ 665us) server_negotiation.cc:299] Negotiation successful
0811 11:54:56.364230 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":131,"threads_started":1}
I20260811 11:54:56.366952 6765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3758e78be88f4e10b3ab6f0f17622a94" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 37243 } }
I20260811 11:54:56.367225 6765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:56.367277 6765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3758e78be88f4e10b3ab6f0f17622a94, State: Initialized, Role: FOLLOWER
I20260811 11:54:56.367774 6765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3758e78be88f4e10b3ab6f0f17622a94" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 37243 } }
I20260811 11:54:56.367923 6765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:54:56.367996 6765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:54:56.368101 6765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:56.369342 6765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3758e78be88f4e10b3ab6f0f17622a94" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 37243 } }
I20260811 11:54:56.369650 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:54:56.369760 6765 leader_election.cc:302] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3758e78be88f4e10b3ab6f0f17622a94; no voters:
I20260811 11:54:56.370128 6765 leader_election.cc:288] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:54:56.370225 6773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:56.370469 6773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94 [term 1 LEADER]: Becoming Leader. State: Replica: 3758e78be88f4e10b3ab6f0f17622a94, State: Running, Role: LEADER
I20260811 11:54:56.370826 6773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3758e78be88f4e10b3ab6f0f17622a94" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 37243 } }
I20260811 11:54:56.371974 6765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:54:56.372128 6773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3758e78be88f4e10b3ab6f0f17622a94. Latest consensus state: current_term: 1 leader_uuid: "3758e78be88f4e10b3ab6f0f17622a94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3758e78be88f4e10b3ab6f0f17622a94" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 37243 } } }
I20260811 11:54:56.372314 6773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:56.372746 6788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:54:56.373056 6775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3758e78be88f4e10b3ab6f0f17622a94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3758e78be88f4e10b3ab6f0f17622a94" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 37243 } } }
I20260811 11:54:56.373154 6775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:56.376389 6788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:54:56.382038 6788 catalog_manager.cc:1347] Generated new cluster ID: da541714f7554d96ba102ee83720e3db
I20260811 11:54:56.382113 6788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:54:56.386656 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:56.372477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44853 (local address 127.3.221.126:37243)
0811 11:54:56.372598 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:56.372603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:56.375015 (+ 2412us) server_negotiation.cc:407] Connection header received
0811 11:54:56.375418 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:56.375421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:56.375479 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:56.375575 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:56.376442 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:56.377609 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:56.379132 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:56.379608 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:56.379929 (+ 321us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:56.382167 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:56.382172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:56.382175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:54:56.382205 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:56.383903 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:56.384545 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:56.384549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:56.384551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:56.384643 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:54:56.385065 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:54:56.385069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:54:56.385070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:54:56.385349 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:56.385501 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:56.386360 (+ 859us) server_negotiation.cc:299] Negotiation successful
0811 11:54:56.386472 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 11:54:56.414065 6788 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:54:56.415066 6788 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:54:56.432011 6788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3758e78be88f4e10b3ab6f0f17622a94: Generated new TSK 0
I20260811 11:54:56.433086 6788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:54:56.444129 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 6647
W20260811 11:54:56.453997 6628 connection.cc:535] client connection to 127.3.221.126:37243 recv error: Network error: failed to read from TLS socket (remote: 127.3.221.126:37243): Connection reset by peer (error 104)
2026-08-11T11:54:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1638 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:54:56 dist-test-slave-bndk krb5kdc[6822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:54:56 dist-test-slave-bndk krb5kdc[6822](info): set up 2 sockets
Aug 11 11:54:56 dist-test-slave-bndk krb5kdc[6822](info): commencing operation
krb5kdc: starting...
W20260811 11:54:57.624876 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:54:57 dist-test-slave-bndk krb5kdc[6822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:54:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:54:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:54:57.762488 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:33889
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:33889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:57.881295 6845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:57.881597 6845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:57.881657 6845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:57.885818 6845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:54:57.885900 6845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:57.885926 6845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:57.885949 6845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:54:57.885978 6845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:54:57.891755 6845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:33889
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:33889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:57.892944 6845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:57.894050 6845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:54:57.902900 6845 server_base.cc:1034] running on GCE node
W20260811 11:54:57.902834 6852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:57.902834 6851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:57.902850 6855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:57.903743 6845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:57.904920 6845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:57.908211 6845 hybrid_clock.cc:648] HybridClock initialized: now 1786449297908168 us; error 71 us; skew 500 ppm
Aug 11 11:54:57 dist-test-slave-bndk krb5kdc[6822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449297, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:57.915409 6845 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:54:57.916630 6845 webserver.cc:460] Webserver started at http://127.3.221.126:40741/ using document root <none> and password file <none>
I20260811 11:54:57.917516 6845 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:57.917624 6845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:57.917904 6845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:57.920409 6845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "b020db0d26eb438f806da72cdb4ecb8d"
format_stamp: "Formatted at 2026-08-11 11:54:57 on dist-test-slave-bndk"
server_key: "a0f4e22b565950271b31da9a575dcb41"
server_key_iv: "ca13ab41f61cedd431b92f29d6a231fb"
server_key_version: "encryptionkey@0"
I20260811 11:54:57.921030 6845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "b020db0d26eb438f806da72cdb4ecb8d"
format_stamp: "Formatted at 2026-08-11 11:54:57 on dist-test-slave-bndk"
server_key: "a0f4e22b565950271b31da9a575dcb41"
server_key_iv: "ca13ab41f61cedd431b92f29d6a231fb"
server_key_version: "encryptionkey@0"
I20260811 11:54:57.924999 6845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 11:54:57.927706 6868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:57.929339 6845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:54:57.929467 6845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "b020db0d26eb438f806da72cdb4ecb8d"
format_stamp: "Formatted at 2026-08-11 11:54:57 on dist-test-slave-bndk"
server_key: "a0f4e22b565950271b31da9a575dcb41"
server_key_iv: "ca13ab41f61cedd431b92f29d6a231fb"
server_key_version: "encryptionkey@0"
I20260811 11:54:57.929587 6845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:57.958540 6845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:57.961724 6845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:57.961913 6845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:57.969681 6845 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:33889
I20260811 11:54:57.969708 6954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:33889 every 8 connection(s)
I20260811 11:54:57.970752 6845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:54:57.973945 6956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:57.979760 6956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d: Bootstrap starting.
I20260811 11:54:57.981022 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 6845
I20260811 11:54:57.981174 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:54:57.981504 3957 external_mini_cluster.cc:1428] Setting key 8adec8017c737a0d311bf0b07d77e16b
I20260811 11:54:57.982748 6956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:57.983601 6956 log.cc:824] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:57.985760 6956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d: No bootstrap required, opened a new log
I20260811 11:54:57.989466 6956 raft_consensus.cc:348] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b020db0d26eb438f806da72cdb4ecb8d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33889 } }
I20260811 11:54:57.989722 6956 raft_consensus.cc:374] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:57.989799 6956 raft_consensus.cc:729] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b020db0d26eb438f806da72cdb4ecb8d, State: Initialized, Role: FOLLOWER
I20260811 11:54:57.990276 6956 consensus_queue.cc:260] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b020db0d26eb438f806da72cdb4ecb8d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33889 } }
I20260811 11:54:57.990425 6956 raft_consensus.cc:388] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:54:57.990494 6956 raft_consensus.cc:482] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:54:57.990597 6956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:57.990777 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:57.982736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59543 (local address 127.3.221.126:33889)
0811 11:54:57.983266 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:57.983282 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:57.983320 (+ 38us) server_negotiation.cc:407] Connection header received
0811 11:54:57.984096 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:57.984123 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:57.984549 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:57.984879 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:57.985345 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:57.986358 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:57.987236 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:57.987718 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:57.987877 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:57.988858 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:57.988863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:57.988877 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:57.989321 (+ 444us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:57.989385 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:57.989410 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:57.989528 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:57.989720 (+ 192us) server_negotiation.cc:299] Negotiation successful
0811 11:54:57.989883 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":171,"threads_started":1}
I20260811 11:54:57.991870 6956 raft_consensus.cc:504] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b020db0d26eb438f806da72cdb4ecb8d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33889 } }
I20260811 11:54:57.992242 6956 leader_election.cc:302] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b020db0d26eb438f806da72cdb4ecb8d; no voters:
I20260811 11:54:57.992581 6956 leader_election.cc:288] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:54:57.992667 6963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:57.992851 6963 raft_consensus.cc:686] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d [term 1 LEADER]: Becoming Leader. State: Replica: b020db0d26eb438f806da72cdb4ecb8d, State: Running, Role: LEADER
I20260811 11:54:57.993165 6963 consensus_queue.cc:237] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b020db0d26eb438f806da72cdb4ecb8d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33889 } }
I20260811 11:54:57.993952 6956 sys_catalog.cc:564] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:54:57.994206 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:54:57.997092 6966 sys_catalog.cc:455] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b020db0d26eb438f806da72cdb4ecb8d. Latest consensus state: current_term: 1 leader_uuid: "b020db0d26eb438f806da72cdb4ecb8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b020db0d26eb438f806da72cdb4ecb8d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33889 } } }
I20260811 11:54:57.997269 6966 sys_catalog.cc:458] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:57.996850 6964 sys_catalog.cc:455] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b020db0d26eb438f806da72cdb4ecb8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b020db0d26eb438f806da72cdb4ecb8d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33889 } } }
I20260811 11:54:57.997646 6964 sys_catalog.cc:458] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d [sys.catalog]: This master's current role is: LEADER
W20260811 11:54:57.999840 6993 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:54:57.999909 6993 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:54:57.999990 6999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:54:58.000761 6999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:54:58.001948 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:57.996915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46041 (local address 127.3.221.126:33889)
0811 11:54:57.997042 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:57.997046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:57.997493 (+ 447us) server_negotiation.cc:407] Connection header received
0811 11:54:57.997599 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:57.997602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:57.997650 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:57.997721 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:57.998132 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:57.998920 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:58.000071 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:54:58.000413 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:54:58.000521 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:54:58.001450 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:58.001454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:58.001457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:58.001471 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:58.001503 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:58.001517 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:58.001613 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:58.001770 (+ 157us) server_negotiation.cc:299] Negotiation successful
0811 11:54:58.001810 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260811 11:54:58.007001 6999 catalog_manager.cc:1347] Generated new cluster ID: 5f15b892647247118328bdeb54b18a55
I20260811 11:54:58.007091 6999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:54:58.023882 6999 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:54:58.025326 6999 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:54:58.040980 6999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b020db0d26eb438f806da72cdb4ecb8d: Generated new TSK 0
I20260811 11:54:58.042030 6999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:54:58.055447 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 6845
W20260811 11:54:58.075307 6818 connection.cc:535] client connection to 127.3.221.126:33889 recv error: Network error: failed to read from TLS socket (remote: 127.3.221.126:33889): Connection reset by peer (error 104)
2026-08-11T11:54:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1621 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:54:58 dist-test-slave-bndk krb5kdc[7006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:54:58 dist-test-slave-bndk krb5kdc[7006](info): set up 2 sockets
Aug 11 11:54:58 dist-test-slave-bndk krb5kdc[7006](info): commencing operation
krb5kdc: starting...
W20260811 11:54:59.236565 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:54:59 dist-test-slave-bndk krb5kdc[7006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:54:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:54:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:54:59.372447 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:33129
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:33129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:54:59.489620 7027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:54:59.489939 7027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:54:59.489984 7027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:54:59.494104 7027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:54:59.494176 7027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:54:59.494195 7027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:54:59.494211 7027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:54:59.494225 7027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:54:59.500123 7027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:33129
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:33129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:54:59.501363 7027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:54:59.502400 7027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:54:59.512327 7038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:59.512393 7027 server_base.cc:1034] running on GCE node
W20260811 11:54:59.512341 7040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:54:59.512327 7042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:54:59.513159 7027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:54:59.514493 7027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:54:59.515722 7027 hybrid_clock.cc:648] HybridClock initialized: now 1786449299515691 us; error 46 us; skew 500 ppm
Aug 11 11:54:59 dist-test-slave-bndk krb5kdc[7006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449299, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:54:59.518987 7027 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:54:59.519979 7027 webserver.cc:460] Webserver started at http://127.3.221.126:37275/ using document root <none> and password file <none>
I20260811 11:54:59.520562 7027 fs_manager.cc:359] Metadata directory not provided
I20260811 11:54:59.520633 7027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:54:59.520841 7027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:54:59.522995 7027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "550367e2bd2846ee814cb9d7894898a7"
format_stamp: "Formatted at 2026-08-11 11:54:59 on dist-test-slave-bndk"
server_key: "a5b5183b9df528e8435946f8f8dfcae3"
server_key_iv: "d9f6a7a75ce2f95f770a0f02a52f453e"
server_key_version: "encryptionkey@0"
I20260811 11:54:59.523566 7027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "550367e2bd2846ee814cb9d7894898a7"
format_stamp: "Formatted at 2026-08-11 11:54:59 on dist-test-slave-bndk"
server_key: "a5b5183b9df528e8435946f8f8dfcae3"
server_key_iv: "d9f6a7a75ce2f95f770a0f02a52f453e"
server_key_version: "encryptionkey@0"
I20260811 11:54:59.527478 7027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260811 11:54:59.530131 7054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:54:59.531455 7027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 11:54:59.531597 7027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "550367e2bd2846ee814cb9d7894898a7"
format_stamp: "Formatted at 2026-08-11 11:54:59 on dist-test-slave-bndk"
server_key: "a5b5183b9df528e8435946f8f8dfcae3"
server_key_iv: "d9f6a7a75ce2f95f770a0f02a52f453e"
server_key_version: "encryptionkey@0"
I20260811 11:54:59.531709 7027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:54:59.536494 7027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:54:59.539789 7027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:54:59.539978 7027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:54:59.547008 7027 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:33129
I20260811 11:54:59.547041 7148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:33129 every 8 connection(s)
I20260811 11:54:59.547983 7027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:54:59.550390 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 7027
I20260811 11:54:59.550526 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:54:59.550808 3957 external_mini_cluster.cc:1428] Setting key 8f9f3211b7df02c269736cd2d2f5e0c9
I20260811 11:54:59.551014 7150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:54:59.556165 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:59.552153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59469 (local address 127.3.221.126:33129)
0811 11:54:59.552606 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:59.552622 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:59.552660 (+ 38us) server_negotiation.cc:407] Connection header received
0811 11:54:59.553364 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:59.553382 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:59.553677 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:59.553962 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:59.554877 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:59.554882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:59.554895 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:59.554907 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:59.554970 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:59.554994 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:59.555127 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:59.555220 (+ 93us) server_negotiation.cc:299] Negotiation successful
0811 11:54:59.555408 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":159,"threads_started":1}
I20260811 11:54:59.558354 7150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7: Bootstrap starting.
I20260811 11:54:59.559533 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:54:59.560913 7150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7: Neither blocks nor log segments found. Creating new log.
I20260811 11:54:59.561870 7150 log.cc:824] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7: Log is configured to *not* fsync() on all Append() calls
I20260811 11:54:59.563167 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:54:59.561577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44349 (local address 127.3.221.126:33129)
0811 11:54:59.561693 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0811 11:54:59.561698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:54:59.561899 (+ 201us) server_negotiation.cc:407] Connection header received
0811 11:54:59.562018 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:54:59.562021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:54:59.562059 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:54:59.562123 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:54:59.562799 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:54:59.562803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:54:59.562805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:54:59.562817 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:54:59.562846 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:54:59.562856 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:54:59.562929 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 11:54:59.563004 (+ 75us) server_negotiation.cc:299] Negotiation successful
0811 11:54:59.563044 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 11:54:59.564036 7150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7: No bootstrap required, opened a new log
I20260811 11:54:59.567938 7150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "550367e2bd2846ee814cb9d7894898a7" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33129 } }
I20260811 11:54:59.568200 7150 raft_consensus.cc:374] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:54:59.568250 7150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 550367e2bd2846ee814cb9d7894898a7, State: Initialized, Role: FOLLOWER
I20260811 11:54:59.568699 7150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "550367e2bd2846ee814cb9d7894898a7" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33129 } }
I20260811 11:54:59.568831 7150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:54:59.568887 7150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:54:59.568984 7150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:54:59.570024 7150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "550367e2bd2846ee814cb9d7894898a7" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33129 } }
I20260811 11:54:59.570381 7150 leader_election.cc:302] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 550367e2bd2846ee814cb9d7894898a7; no voters:
I20260811 11:54:59.570693 7150 leader_election.cc:288] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:54:59.570858 7169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:54:59.571086 7169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7 [term 1 LEADER]: Becoming Leader. State: Replica: 550367e2bd2846ee814cb9d7894898a7, State: Running, Role: LEADER
I20260811 11:54:59.571409 7169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "550367e2bd2846ee814cb9d7894898a7" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33129 } }
I20260811 11:54:59.572149 7150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:54:59.573309 7172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 550367e2bd2846ee814cb9d7894898a7. Latest consensus state: current_term: 1 leader_uuid: "550367e2bd2846ee814cb9d7894898a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "550367e2bd2846ee814cb9d7894898a7" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33129 } } }
I20260811 11:54:59.573462 7172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:59.573724 7171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "550367e2bd2846ee814cb9d7894898a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "550367e2bd2846ee814cb9d7894898a7" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33129 } } }
I20260811 11:54:59.573833 7171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7 [sys.catalog]: This master's current role is: LEADER
I20260811 11:54:59.573865 7184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:54:59.577114 7184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:54:59.583139 7184 catalog_manager.cc:1347] Generated new cluster ID: 581389f4c5064df8b7c39385d8fe494e
I20260811 11:54:59.583225 7184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:54:59.606566 7184 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:54:59.606978 7184 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:54:59.613063 7184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 550367e2bd2846ee814cb9d7894898a7: Generated new TSK 0
I20260811 11:54:59.613847 7184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:54:59.623910 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 7027
2026-08-11T11:54:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1564 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:54:59 dist-test-slave-bndk krb5kdc[7210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:54:59 dist-test-slave-bndk krb5kdc[7210](info): set up 2 sockets
Aug 11 11:54:59 dist-test-slave-bndk krb5kdc[7210](info): commencing operation
krb5kdc: starting...
W20260811 11:55:00.807103 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:55:00 dist-test-slave-bndk krb5kdc[7210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:55:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:55:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:55:00.945027 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:44881
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:44881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:55:01.068006 7234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:55:01.068349 7234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:55:01.068414 7234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:55:01.072707 7234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:55:01.072784 7234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:55:01.072811 7234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:55:01.072834 7234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:55:01.072855 7234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:55:01.078580 7234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:44881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:44881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:55:01.079771 7234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:55:01.080843 7234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:55:01.090199 7240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:55:01.090189 7241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:55:01.090188 7247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:55:01.090256 7234 server_base.cc:1034] running on GCE node
I20260811 11:55:01.091053 7234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:55:01.092269 7234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:55:01.093564 7234 hybrid_clock.cc:648] HybridClock initialized: now 1786449301093469 us; error 41 us; skew 500 ppm
Aug 11 11:55:01 dist-test-slave-bndk krb5kdc[7210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449301, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:55:01.096675 7234 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:55:01.097746 7234 webserver.cc:460] Webserver started at http://127.3.221.126:36973/ using document root <none> and password file <none>
I20260811 11:55:01.098333 7234 fs_manager.cc:359] Metadata directory not provided
I20260811 11:55:01.098414 7234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:55:01.098634 7234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:55:01.101713 7234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "81d9e8ae1d244c858351380e8fcb54ea"
format_stamp: "Formatted at 2026-08-11 11:55:01 on dist-test-slave-bndk"
server_key: "e05ff4b57156177705d699247032442d"
server_key_iv: "b5615c60ac4d8d1c1c09aad626331788"
server_key_version: "encryptionkey@0"
I20260811 11:55:01.102360 7234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "81d9e8ae1d244c858351380e8fcb54ea"
format_stamp: "Formatted at 2026-08-11 11:55:01 on dist-test-slave-bndk"
server_key: "e05ff4b57156177705d699247032442d"
server_key_iv: "b5615c60ac4d8d1c1c09aad626331788"
server_key_version: "encryptionkey@0"
I20260811 11:55:01.106495 7234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:55:01.109431 7260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:55:01.110874 7234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:55:01.111022 7234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "81d9e8ae1d244c858351380e8fcb54ea"
format_stamp: "Formatted at 2026-08-11 11:55:01 on dist-test-slave-bndk"
server_key: "e05ff4b57156177705d699247032442d"
server_key_iv: "b5615c60ac4d8d1c1c09aad626331788"
server_key_version: "encryptionkey@0"
I20260811 11:55:01.111150 7234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:55:01.123044 7234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:55:01.126607 7234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:55:01.126839 7234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:55:01.134567 7234 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:44881
I20260811 11:55:01.134593 7354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:44881 every 8 connection(s)
I20260811 11:55:01.135632 7234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:55:01.138653 7356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:55:01.143627 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 7234
I20260811 11:55:01.143800 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:55:01.144131 3957 external_mini_cluster.cc:1428] Setting key ca75de9f5b7c3d5d2ffcb30e5a186e07
I20260811 11:55:01.146152 7356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea: Bootstrap starting.
I20260811 11:55:01.149281 7356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea: Neither blocks nor log segments found. Creating new log.
I20260811 11:55:01.150104 7356 log.cc:824] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea: Log is configured to *not* fsync() on all Append() calls
I20260811 11:55:01.152581 7356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea: No bootstrap required, opened a new log
Aug 11 11:55:01 dist-test-slave-bndk krb5kdc[7210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:55:01.157363 7356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d9e8ae1d244c858351380e8fcb54ea" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 44881 } }
I20260811 11:55:01.157696 7356 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:55:01.157758 7356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d9e8ae1d244c858351380e8fcb54ea, State: Initialized, Role: FOLLOWER
I20260811 11:55:01.158308 7356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d9e8ae1d244c858351380e8fcb54ea" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 44881 } }
I20260811 11:55:01.158470 7356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:55:01.158540 7356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:55:01.158638 7356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:55:01.159592 7356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d9e8ae1d244c858351380e8fcb54ea" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 44881 } }
I20260811 11:55:01.159945 7356 leader_election.cc:302] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81d9e8ae1d244c858351380e8fcb54ea; no voters:
I20260811 11:55:01.160316 7356 leader_election.cc:288] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:55:01.160409 7362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:55:01.160629 7362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea [term 1 LEADER]: Becoming Leader. State: Replica: 81d9e8ae1d244c858351380e8fcb54ea, State: Running, Role: LEADER
I20260811 11:55:01.161024 7362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d9e8ae1d244c858351380e8fcb54ea" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 44881 } }
I20260811 11:55:01.161499 7360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:01.146294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57003 (local address 127.3.221.126:44881)
0811 11:55:01.146757 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:01.146773 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:01.146812 (+ 39us) server_negotiation.cc:407] Connection header received
0811 11:55:01.147526 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:01.147543 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:01.147914 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:01.148258 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:01.149101 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:01.150685 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:01.152034 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:01.152897 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:01.153046 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:01.156050 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:01.156059 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:01.156082 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:01.156123 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:01.158168 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:01.158682 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:01.158689 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:01.158695 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:01.158848 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:01.159122 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:01.159126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:01.159128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:01.159574 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:01.159773 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:01.160057 (+ 284us) server_negotiation.cc:299] Negotiation successful
0811 11:55:01.160426 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":121,"threads_started":1}
I20260811 11:55:01.161633 7356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:55:01.162977 7364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81d9e8ae1d244c858351380e8fcb54ea. Latest consensus state: current_term: 1 leader_uuid: "81d9e8ae1d244c858351380e8fcb54ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d9e8ae1d244c858351380e8fcb54ea" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 44881 } } }
I20260811 11:55:01.163105 7364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea [sys.catalog]: This master's current role is: LEADER
I20260811 11:55:01.162949 7363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81d9e8ae1d244c858351380e8fcb54ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d9e8ae1d244c858351380e8fcb54ea" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 44881 } } }
I20260811 11:55:01.163365 7363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea [sys.catalog]: This master's current role is: LEADER
I20260811 11:55:01.164270 7376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:55:01.164885 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:55:01.167526 7376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:55:01.175459 7376 catalog_manager.cc:1347] Generated new cluster ID: 7578da149f7e40c58cdb1a1822320ceb
I20260811 11:55:01.175530 7376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:55:01.177165 7360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:01.167460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34329 (local address 127.3.221.126:44881)
0811 11:55:01.167569 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:01.167573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:01.167587 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:55:01.167630 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:01.167633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:01.167678 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:01.167763 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:01.168710 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:01.169624 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:01.170818 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:01.171162 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:01.171370 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:01.173634 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:01.173638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:01.173641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:01.173670 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:01.175260 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:01.175857 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:01.175860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:01.175862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:01.175955 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:01.176249 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:01.176252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:01.176254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:01.176494 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:01.176613 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:01.176887 (+ 274us) server_negotiation.cc:299] Negotiation successful
0811 11:55:01.177009 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 11:55:01.208427 7376 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:55:01.209753 7376 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:55:01.221138 7376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81d9e8ae1d244c858351380e8fcb54ea: Generated new TSK 0
I20260811 11:55:01.221935 7376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:55:01.233246 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 7234
2026-08-11T11:55:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1622 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:55:01 dist-test-slave-bndk krb5kdc[7401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:55:01 dist-test-slave-bndk krb5kdc[7401](info): set up 2 sockets
Aug 11 11:55:01 dist-test-slave-bndk krb5kdc[7401](info): commencing operation
krb5kdc: starting...
W20260811 11:55:02.445466 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:55:02 dist-test-slave-bndk krb5kdc[7401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:55:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:55:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:55:02.580955 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:46373
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:41471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:46373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:55:02.709815 7418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:55:02.710245 7418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:55:02.710332 7418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:55:02.715328 7418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:55:02.715412 7418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:55:02.715440 7418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:55:02.715464 7418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:55:02.715494 7418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:55:02.723858 7418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:41471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:46373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:46373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:55:02.725303 7418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:55:02.726395 7418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:55:02.735113 7426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:55:02.735116 7429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:55:02.735160 7431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:55:02.735226 7418 server_base.cc:1034] running on GCE node
I20260811 11:55:02.735935 7418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:55:02.737052 7418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:55:02.738245 7418 hybrid_clock.cc:648] HybridClock initialized: now 1786449302738232 us; error 50 us; skew 500 ppm
Aug 11 11:55:02 dist-test-slave-bndk krb5kdc[7401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449302, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:55:02.741106 7418 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:55:02.742089 7418 webserver.cc:460] Webserver started at http://127.3.221.126:46861/ using document root <none> and password file <none>
I20260811 11:55:02.742652 7418 fs_manager.cc:359] Metadata directory not provided
I20260811 11:55:02.742712 7418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:55:02.742892 7418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:55:02.745023 7418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "1a169aa513f34828956985ca4a15f0ab"
format_stamp: "Formatted at 2026-08-11 11:55:02 on dist-test-slave-bndk"
server_key: "f755f2b17b0a5ee712e3262dc1eb48c5"
server_key_iv: "fae0893378315867f5b80c5ae644dba6"
server_key_version: "encryptionkey@0"
I20260811 11:55:02.745635 7418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "1a169aa513f34828956985ca4a15f0ab"
format_stamp: "Formatted at 2026-08-11 11:55:02 on dist-test-slave-bndk"
server_key: "f755f2b17b0a5ee712e3262dc1eb48c5"
server_key_iv: "fae0893378315867f5b80c5ae644dba6"
server_key_version: "encryptionkey@0"
I20260811 11:55:02.749569 7418 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:55:02.752125 7448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:55:02.753531 7418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 11:55:02.753659 7418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "1a169aa513f34828956985ca4a15f0ab"
format_stamp: "Formatted at 2026-08-11 11:55:02 on dist-test-slave-bndk"
server_key: "f755f2b17b0a5ee712e3262dc1eb48c5"
server_key_iv: "fae0893378315867f5b80c5ae644dba6"
server_key_version: "encryptionkey@0"
I20260811 11:55:02.753782 7418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:55:02.780959 7418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:55:02.784298 7418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:55:02.784466 7418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:55:02.791778 7418 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:46373
I20260811 11:55:02.791801 7541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:46373 every 8 connection(s)
I20260811 11:55:02.792778 7418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:55:02.793388 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 7418
I20260811 11:55:02.793550 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:55:02.793833 3957 external_mini_cluster.cc:1428] Setting key dd7fd89b512074cd38c90c07ebc162ef
I20260811 11:55:02.796360 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:55:02.802403 7542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab: Bootstrap starting.
I20260811 11:55:02.803665 7547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:02.795414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36855 (local address 127.3.221.126:46373)
0811 11:55:02.796070 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:02.796093 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:02.796142 (+ 49us) server_negotiation.cc:407] Connection header received
0811 11:55:02.796908 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:02.796928 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:02.797407 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:02.797781 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:02.798210 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:02.799247 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:02.800159 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:02.800773 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:02.800963 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:02.801819 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:02.801824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:02.801838 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:55:02.802268 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:02.802328 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:55:02.802354 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:02.802447 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:02.802594 (+ 147us) server_negotiation.cc:299] Negotiation successful
0811 11:55:02.802899 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":165,"threads_started":1}
I20260811 11:55:02.805009 7542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab: Neither blocks nor log segments found. Creating new log.
I20260811 11:55:02.806061 7542 log.cc:824] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab: Log is configured to *not* fsync() on all Append() calls
I20260811 11:55:02.807065 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:55:02.808301 7542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab: No bootstrap required, opened a new log
I20260811 11:55:02.812166 7542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a169aa513f34828956985ca4a15f0ab" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46373 } }
I20260811 11:55:02.812433 7542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:55:02.812522 7542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a169aa513f34828956985ca4a15f0ab, State: Initialized, Role: FOLLOWER
I20260811 11:55:02.812952 7547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:02.809076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47381 (local address 127.3.221.126:46373)
0811 11:55:02.809200 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:02.809205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:02.809443 (+ 238us) server_negotiation.cc:407] Connection header received
0811 11:55:02.809550 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:02.809553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:02.809599 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:02.809661 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:02.809937 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:02.810544 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:02.811382 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:02.811627 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:02.811719 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:02.812530 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:02.812534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:02.812537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:55:02.812553 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:02.812586 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:55:02.812600 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:02.812676 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:02.812757 (+ 81us) server_negotiation.cc:299] Negotiation successful
0811 11:55:02.812817 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 11:55:02.813057 7542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a169aa513f34828956985ca4a15f0ab" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46373 } }
I20260811 11:55:02.813256 7542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:55:02.813334 7542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:55:02.813441 7542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:55:02.814738 7542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a169aa513f34828956985ca4a15f0ab" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46373 } }
I20260811 11:55:02.815083 7542 leader_election.cc:302] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a169aa513f34828956985ca4a15f0ab; no voters:
I20260811 11:55:02.815373 7542 leader_election.cc:288] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:55:02.815505 7559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:55:02.815730 7559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab [term 1 LEADER]: Becoming Leader. State: Replica: 1a169aa513f34828956985ca4a15f0ab, State: Running, Role: LEADER
I20260811 11:55:02.816079 7559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a169aa513f34828956985ca4a15f0ab" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46373 } }
I20260811 11:55:02.816627 7542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:55:02.817989 7560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a169aa513f34828956985ca4a15f0ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a169aa513f34828956985ca4a15f0ab" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46373 } } }
I20260811 11:55:02.818146 7560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab [sys.catalog]: This master's current role is: LEADER
I20260811 11:55:02.818022 7564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a169aa513f34828956985ca4a15f0ab. Latest consensus state: current_term: 1 leader_uuid: "1a169aa513f34828956985ca4a15f0ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a169aa513f34828956985ca4a15f0ab" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 46373 } } }
I20260811 11:55:02.818462 7564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab [sys.catalog]: This master's current role is: LEADER
I20260811 11:55:02.819047 7572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:55:02.822683 7572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:55:02.828467 7572 catalog_manager.cc:1347] Generated new cluster ID: ea02376f7d5a4c9b909d3766a89aabe5
I20260811 11:55:02.828558 7572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:55:02.837473 7572 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:55:02.838438 7572 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:55:02.848047 7572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a169aa513f34828956985ca4a15f0ab: Generated new TSK 0
I20260811 11:55:02.848783 7572 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:55:02.854730 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 7418
2026-08-11T11:55:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1607 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:55:02 dist-test-slave-bndk krb5kdc[7587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:55:02 dist-test-slave-bndk krb5kdc[7587](info): set up 2 sockets
Aug 11 11:55:02 dist-test-slave-bndk krb5kdc[7587](info): commencing operation
krb5kdc: starting...
W20260811 11:55:04.054620 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.170s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:55:04 dist-test-slave-bndk krb5kdc[7587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:55:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:55:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:55:04.196545 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:37507
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:35827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:37507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:55:04.319298 7602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:55:04.319656 7602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:55:04.319721 7602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:55:04.324638 7602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:55:04.324744 7602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:55:04.324774 7602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:55:04.324795 7602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:55:04.324817 7602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:55:04.330746 7602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:35827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:37507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:37507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:55:04.332006 7602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:55:04.333132 7602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:55:04.342336 7611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:55:04.342293 7613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:55:04.342319 7610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:55:04.342618 7602 server_base.cc:1034] running on GCE node
I20260811 11:55:04.343215 7602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:55:04.344432 7602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:55:04.345630 7602 hybrid_clock.cc:648] HybridClock initialized: now 1786449304345615 us; error 35 us; skew 500 ppm
Aug 11 11:55:04 dist-test-slave-bndk krb5kdc[7587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449304, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:55:04.354176 7602 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:55:04.355583 7602 webserver.cc:460] Webserver started at http://127.3.221.126:45907/ using document root <none> and password file <none>
I20260811 11:55:04.356534 7602 fs_manager.cc:359] Metadata directory not provided
I20260811 11:55:04.356652 7602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:55:04.356958 7602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:55:04.360404 7602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "c3a004a8d48141008045b83786d37a84"
format_stamp: "Formatted at 2026-08-11 11:55:04 on dist-test-slave-bndk"
server_key: "0359aea1671b84449e021aff637a3c79"
server_key_iv: "c21f49af5432f8a98b2fb26ee310779d"
server_key_version: "encryptionkey@0"
I20260811 11:55:04.361397 7602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "c3a004a8d48141008045b83786d37a84"
format_stamp: "Formatted at 2026-08-11 11:55:04 on dist-test-slave-bndk"
server_key: "0359aea1671b84449e021aff637a3c79"
server_key_iv: "c21f49af5432f8a98b2fb26ee310779d"
server_key_version: "encryptionkey@0"
I20260811 11:55:04.367128 7602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260811 11:55:04.370260 7630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:55:04.371743 7602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 11:55:04.371908 7602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "c3a004a8d48141008045b83786d37a84"
format_stamp: "Formatted at 2026-08-11 11:55:04 on dist-test-slave-bndk"
server_key: "0359aea1671b84449e021aff637a3c79"
server_key_iv: "c21f49af5432f8a98b2fb26ee310779d"
server_key_version: "encryptionkey@0"
I20260811 11:55:04.372040 7602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:55:04.377009 7602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:55:04.380718 7602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:55:04.380921 7602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:55:04.388798 7602 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:37507
I20260811 11:55:04.388856 7718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:37507 every 8 connection(s)
I20260811 11:55:04.389883 7602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:55:04.392972 7719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:55:04.395802 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 7602
I20260811 11:55:04.395991 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:55:04.396390 3957 external_mini_cluster.cc:1428] Setting key 2973848b4d31ae6eb42830d549501653
I20260811 11:55:04.399967 7719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84: Bootstrap starting.
I20260811 11:55:04.402915 7719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84: Neither blocks nor log segments found. Creating new log.
I20260811 11:55:04.403350 7723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:04.398159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40851 (local address 127.3.221.126:37507)
0811 11:55:04.398724 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:04.398746 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:04.398806 (+ 60us) server_negotiation.cc:407] Connection header received
0811 11:55:04.399701 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:04.399726 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:04.400149 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:04.400545 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:04.401807 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:04.401813 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:04.401829 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:55:04.401843 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:04.401911 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:55:04.401940 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:04.402098 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:04.402206 (+ 108us) server_negotiation.cc:299] Negotiation successful
0811 11:55:04.402368 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":231,"threads_started":1}
I20260811 11:55:04.403843 7719 log.cc:824] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84: Log is configured to *not* fsync() on all Append() calls
I20260811 11:55:04.406081 7719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84: No bootstrap required, opened a new log
I20260811 11:55:04.406431 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:55:04.410168 7723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:04.408415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46653 (local address 127.3.221.126:37507)
0811 11:55:04.408532 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:04.408536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:04.408678 (+ 142us) server_negotiation.cc:407] Connection header received
0811 11:55:04.408805 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:04.408807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:04.408838 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:04.408901 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:04.409713 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:04.409716 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:04.409718 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 11:55:04.409729 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:04.409762 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 11:55:04.409773 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:04.409852 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:04.409995 (+ 143us) server_negotiation.cc:299] Negotiation successful
0811 11:55:04.410037 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 11:55:04.410315 7719 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a004a8d48141008045b83786d37a84" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 37507 } }
I20260811 11:55:04.410640 7719 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 11:55:04.410732 7719 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a004a8d48141008045b83786d37a84, State: Initialized, Role: FOLLOWER
I20260811 11:55:04.411238 7719 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a004a8d48141008045b83786d37a84" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 37507 } }
I20260811 11:55:04.411422 7719 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 11:55:04.411487 7719 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 11:55:04.411595 7719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:55:04.412612 7719 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a004a8d48141008045b83786d37a84" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 37507 } }
I20260811 11:55:04.412982 7719 leader_election.cc:302] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3a004a8d48141008045b83786d37a84; no voters:
I20260811 11:55:04.413341 7719 leader_election.cc:288] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 11:55:04.413446 7733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:55:04.413689 7733 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84 [term 1 LEADER]: Becoming Leader. State: Replica: c3a004a8d48141008045b83786d37a84, State: Running, Role: LEADER
I20260811 11:55:04.414053 7733 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a004a8d48141008045b83786d37a84" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 37507 } }
I20260811 11:55:04.414628 7719 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:55:04.415723 7735 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3a004a8d48141008045b83786d37a84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a004a8d48141008045b83786d37a84" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 37507 } } }
I20260811 11:55:04.415861 7735 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84 [sys.catalog]: This master's current role is: LEADER
I20260811 11:55:04.416199 7736 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3a004a8d48141008045b83786d37a84. Latest consensus state: current_term: 1 leader_uuid: "c3a004a8d48141008045b83786d37a84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a004a8d48141008045b83786d37a84" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 37507 } } }
I20260811 11:55:04.416347 7736 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84 [sys.catalog]: This master's current role is: LEADER
I20260811 11:55:04.416463 7744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:55:04.420027 7744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:55:04.426402 7744 catalog_manager.cc:1347] Generated new cluster ID: b84f4695b8c741e8bac7349dd625544e
I20260811 11:55:04.426501 7744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:55:04.462397 7744 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:55:04.462774 7744 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:55:04.468015 7744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3a004a8d48141008045b83786d37a84: Generated new TSK 0
I20260811 11:55:04.468819 7744 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:55:04.482537 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 7602
2026-08-11T11:55:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1628 ms)
[----------] 18 tests from AuthTokenIssuingTest (28998 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:55:04 dist-test-slave-bndk krb5kdc[7762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:55:04 dist-test-slave-bndk krb5kdc[7762](info): set up 2 sockets
Aug 11 11:55:04 dist-test-slave-bndk krb5kdc[7762](info): commencing operation
krb5kdc: starting...
W20260811 11:55:05.676661 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:55:05 dist-test-slave-bndk krb5kdc[7762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:55:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:55:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:55:05.816725 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:33983
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:33983,127.3.221.125:44917,127.3.221.124:39311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:55:05.938256 7784 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 11:55:05.938591 7784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:55:05.938627 7784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:55:05.938694 7784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:55:05.942852 7784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:55:05.942927 7784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:55:05.942955 7784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:55:05.942976 7784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:55:05.943006 7784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:55:05.949570 7784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:33983,127.3.221.125:44917,127.3.221.124:39311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:33983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:55:05.950984 7784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:55:05.952164 7784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:55:05.961575 7799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:55:05.961706 7793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:55:05.961576 7796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:55:05.962014 7784 server_base.cc:1034] running on GCE node
I20260811 11:55:05.962625 7784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:55:05.963805 7784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:55:05.965003 7784 hybrid_clock.cc:648] HybridClock initialized: now 1786449305964973 us; error 43 us; skew 500 ppm
Aug 11 11:55:05 dist-test-slave-bndk krb5kdc[7762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449305, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:55:05.968055 7784 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:55:05.969075 7784 webserver.cc:460] Webserver started at http://127.3.221.126:44839/ using document root <none> and password file <none>
I20260811 11:55:05.969714 7784 fs_manager.cc:359] Metadata directory not provided
I20260811 11:55:05.969782 7784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:55:05.969977 7784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:55:05.972358 7784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "5ffc12eb3de74815bfb79cd3d06944de"
format_stamp: "Formatted at 2026-08-11 11:55:05 on dist-test-slave-bndk"
server_key: "eda07ffb9d1fdd83dad657d35c9fec85"
server_key_iv: "b0000338084bdf0a3f25dde206b9d0b5"
server_key_version: "encryptionkey@0"
I20260811 11:55:05.972975 7784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "5ffc12eb3de74815bfb79cd3d06944de"
format_stamp: "Formatted at 2026-08-11 11:55:05 on dist-test-slave-bndk"
server_key: "eda07ffb9d1fdd83dad657d35c9fec85"
server_key_iv: "b0000338084bdf0a3f25dde206b9d0b5"
server_key_version: "encryptionkey@0"
I20260811 11:55:05.977134 7784 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260811 11:55:05.980046 7815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:55:05.981686 7784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260811 11:55:05.981820 7784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "5ffc12eb3de74815bfb79cd3d06944de"
format_stamp: "Formatted at 2026-08-11 11:55:05 on dist-test-slave-bndk"
server_key: "eda07ffb9d1fdd83dad657d35c9fec85"
server_key_iv: "b0000338084bdf0a3f25dde206b9d0b5"
server_key_version: "encryptionkey@0"
I20260811 11:55:05.981932 7784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:55:06.005810 7784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:55:06.009089 7784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:55:06.009325 7784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:55:06.016950 7784 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:33983
I20260811 11:55:06.017270 7903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:33983 every 8 connection(s)
I20260811 11:55:06.018088 7784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:55:06.022063 7904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:55:06.025557 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 7784
I20260811 11:55:06.025691 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:55:06.025972 3957 external_mini_cluster.cc:1428] Setting key c78a55d1b735f7a9f0fc7df976b5c6af
I20260811 11:55:06.026218 7904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33983 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.221.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.125@KRBTEST.COM" created.
Aug 11 11:55:06 dist-test-slave-bndk krb5kdc[7762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449305, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:55:06.044862 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.028618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.3.221.126:33983)
0811 11:55:06.029261 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.029273 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.029382 (+ 109us) server_negotiation.cc:407] Connection header received
0811 11:55:06.031623 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.031630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.031925 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.032145 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.033099 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.034567 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.035715 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.036119 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.036252 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.039982 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:06.039990 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:06.040005 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:06.040030 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:06.041688 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.042168 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.042173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.042174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.042283 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.042578 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.042581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.042583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.043005 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:06.043133 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:06.043515 (+ 382us) server_negotiation.cc:299] Negotiation successful
0811 11:55:06.043869 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":143,"server-negotiator.queue_time_us":484,"spinlock_wait_cycles":20352,"thread_start_us":177,"threads_started":1}
I20260811 11:55:06.044860 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.028618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:33983 (local address 127.0.0.1:58338)
0811 11:55:06.029163 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.029252 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0811 11:55:06.030912 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:55:06.032210 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:55:06.032231 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:55:06.032633 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:55:06.032913 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:06.032929 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.034735 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.034738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:06.035585 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:06.035596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.036300 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.036303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:06.036413 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.037019 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:55:06.037038 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:55:06.039806 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:55:06.041799 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:06.041807 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:06.041819 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:06.042059 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:06.042372 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:06.042375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:06.042377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:06.042493 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:06.043120 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:55:06.043126 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:55:06.043271 (+ 145us) client_negotiation.cc:769] Sending connection context
0811 11:55:06.043430 (+ 159us) client_negotiation.cc:240] Negotiation successful
0811 11:55:06.043809 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":1,"thread_start_us":148,"threads_started":1}
I20260811 11:55:06.047580 7904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:55:06.048470 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.048143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:44917 (local address 127.0.0.1:45740)
0811 11:55:06.048305 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.048394 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260811 11:55:06.048682 7821 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.221.125:44917: connect: Connection refused (error 111)
I20260811 11:55:06.049544 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.049342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:44917 (local address 127.0.0.1:45748)
0811 11:55:06.049443 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.049487 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260811 11:55:06.049670 7904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.125:44917: Network error: Client connection negotiation failed: client connection to 127.3.221.125:44917: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.125.keytab.
Entry for principal kudu/127.3.221.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.125.keytab.
I20260811 11:55:06.060389 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.125:44917
--webserver_interface=127.3.221.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.125.keytab
--principal=kudu/127.3.221.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:33983,127.3.221.125:44917,127.3.221.124:39311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:55:06.098970 7904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } attempt: 1
I20260811 11:55:06.100124 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.099808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:44917 (local address 127.0.0.1:45764)
0811 11:55:06.099974 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.100040 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260811 11:55:06.100817 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.100602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:44917 (local address 127.0.0.1:45774)
0811 11:55:06.100703 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.100756 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260811 11:55:06.100936 7904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.125:44917: Network error: Client connection negotiation failed: client connection to 127.3.221.125:44917: connect: Connection refused (error 111)
I20260811 11:55:06.169147 7904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } attempt: 2
I20260811 11:55:06.170035 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.169726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:44917 (local address 127.0.0.1:45786)
0811 11:55:06.169892 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.169953 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260811 11:55:06.170811 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.170604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:44917 (local address 127.0.0.1:45794)
0811 11:55:06.170704 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.170751 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260811 11:55:06.170928 7904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.125:44917: Network error: Client connection negotiation failed: client connection to 127.3.221.125:44917: connect: Connection refused (error 111)
W20260811 11:55:06.182986 7915 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 11:55:06.183328 7915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:55:06.183368 7915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:55:06.183434 7915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:55:06.187891 7915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:55:06.187970 7915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:55:06.187997 7915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:55:06.188020 7915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:55:06.188046 7915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:55:06.194380 7915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:33983,127.3.221.125:44917,127.3.221.124:39311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.125.keytab
--principal=kudu/127.3.221.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.125:44917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.221.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:55:06.195591 7915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:55:06.196668 7915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:55:06.205821 7915 server_base.cc:1034] running on GCE node
W20260811 11:55:06.205802 7921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:55:06.205806 7920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:55:06.206010 7925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:55:06.206661 7915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:55:06.207876 7915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:55:06.209189 7915 hybrid_clock.cc:648] HybridClock initialized: now 1786449306209124 us; error 62 us; skew 500 ppm
Aug 11 11:55:06 dist-test-slave-bndk krb5kdc[7762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:55:06.212990 7915 init.cc:377] Logged in from keytab as kudu/127.3.221.125@KRBTEST.COM (short username kudu)
I20260811 11:55:06.214159 7915 webserver.cc:460] Webserver started at http://127.3.221.125:42773/ using document root <none> and password file <none>
I20260811 11:55:06.214768 7915 fs_manager.cc:359] Metadata directory not provided
I20260811 11:55:06.214840 7915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:55:06.215162 7915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:55:06.217530 7915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-1/data/instance:
uuid: "8215aa3e775b490195f859dfcae15372"
format_stamp: "Formatted at 2026-08-11 11:55:06 on dist-test-slave-bndk"
server_key: "640687c8b2b52442715652f80f399438"
server_key_iv: "1079a82dbdb665840d2a094d609f51c9"
server_key_version: "encryptionkey@0"
I20260811 11:55:06.218140 7915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-1/wal/instance:
uuid: "8215aa3e775b490195f859dfcae15372"
format_stamp: "Formatted at 2026-08-11 11:55:06 on dist-test-slave-bndk"
server_key: "640687c8b2b52442715652f80f399438"
server_key_iv: "1079a82dbdb665840d2a094d609f51c9"
server_key_version: "encryptionkey@0"
I20260811 11:55:06.222329 7915 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 11:55:06.225044 7936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:55:06.226432 7915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 11:55:06.226557 7915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-1/wal
uuid: "8215aa3e775b490195f859dfcae15372"
format_stamp: "Formatted at 2026-08-11 11:55:06 on dist-test-slave-bndk"
server_key: "640687c8b2b52442715652f80f399438"
server_key_iv: "1079a82dbdb665840d2a094d609f51c9"
server_key_version: "encryptionkey@0"
I20260811 11:55:06.226686 7915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:55:06.249467 7915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:55:06.252736 7915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:55:06.252943 7915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:55:06.260921 7915 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.125:44917
I20260811 11:55:06.261270 8023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.125:44917 every 8 connection(s)
I20260811 11:55:06.262015 7915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-1/data/info.pb
I20260811 11:55:06.262094 7904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } attempt: 3
I20260811 11:55:06.265422 8025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:55:06.269347 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 7915
I20260811 11:55:06.269501 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-1/wal/instance
I20260811 11:55:06.269776 3957 external_mini_cluster.cc:1428] Setting key 4e2cade2989f0e685b7c78d22513be12
Aug 11 11:55:06 dist-test-slave-bndk krb5kdc[7762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449305, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for kudu/127.3.221.125@KRBTEST.COM
I20260811 11:55:06.270377 8025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33983 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:55:06.275550 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.262619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:44917 (local address 127.0.0.1:45796)
0811 11:55:06.262761 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.262779 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 11:55:06.262883 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:55:06.265296 (+ 2413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:55:06.265300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:55:06.265522 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:55:06.265672 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:06.265681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.266952 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.266956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:06.267741 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:06.267756 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.268542 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.268547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:06.268670 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.269311 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:55:06.269326 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:55:06.270881 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:55:06.273504 (+ 2623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:06.273508 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:06.273510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:06.273776 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:06.274194 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:06.274197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:06.274199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:06.274295 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:06.275002 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:55:06.275005 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:55:06.275113 (+ 108us) client_negotiation.cc:769] Sending connection context
0811 11:55:06.275196 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 11:55:06.275360 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260811 11:55:06.277086 8028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.263298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45796 (local address 127.3.221.125:44917)
0811 11:55:06.263831 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.263851 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.263889 (+ 38us) server_negotiation.cc:407] Connection header received
0811 11:55:06.264595 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.264613 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.264957 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.265292 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.265810 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.266801 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.267897 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.268382 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.268512 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.271150 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:06.271157 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:06.271176 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:06.271211 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:06.273357 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.273934 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.273940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.273946 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.274100 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.274386 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.274389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.274391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.274874 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:06.275791 (+ 917us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:06.275974 (+ 183us) server_negotiation.cc:299] Negotiation successful
0811 11:55:06.276295 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":390,"thread_start_us":149,"threads_started":1}
I20260811 11:55:06.279384 7904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.221.124@KRBTEST.COM; defaulting to no policy
Aug 11 11:55:06 dist-test-slave-bndk krb5kdc[7762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.125@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:55:06.280277 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.279948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:39311 (local address 127.0.0.1:38048)
0811 11:55:06.280141 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.280195 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260811 11:55:06.281400 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.281055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:39311 (local address 127.0.0.1:38054)
0811 11:55:06.281270 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.281328 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260811 11:55:06.281567 7904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.124:39311: Network error: Client connection negotiation failed: client connection to 127.3.221.124:39311: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.124@KRBTEST.COM" created.
I20260811 11:55:06.284682 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.273667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58354 (local address 127.3.221.126:33983)
0811 11:55:06.273787 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.273791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.273803 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:55:06.273837 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.273840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.273888 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.273964 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.274983 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.276157 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.277803 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.278157 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.278264 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.281142 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:06.281147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:06.281150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:06.281180 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:06.282718 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.283264 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.283269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.283270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.283366 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.283726 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.283729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.283730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.283923 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:06.284004 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:06.284460 (+ 456us) server_negotiation.cc:299] Negotiation successful
0811 11:55:06.284569 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 11:55:06.284762 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.272682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:33983 (local address 127.0.0.1:58354)
0811 11:55:06.273196 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.273278 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0811 11:55:06.273421 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:55:06.274120 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:55:06.274127 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:55:06.274600 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:55:06.274841 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:06.274850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.276304 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.276308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:06.277327 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:06.277341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.278381 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.278384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:06.278497 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.279021 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:55:06.279033 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:55:06.280948 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:55:06.282861 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:06.282873 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:06.282892 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:06.283162 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:06.283497 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:06.283501 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:06.283503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:06.283616 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:06.284028 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:55:06.284035 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:55:06.284188 (+ 153us) client_negotiation.cc:769] Sending connection context
0811 11:55:06.284360 (+ 172us) client_negotiation.cc:240] Negotiation successful
0811 11:55:06.284567 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":46,"thread_start_us":160,"threads_started":1}
I20260811 11:55:06.286123 8025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:55:06 dist-test-slave-bndk krb5kdc[7762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.125@KRBTEST.COM for kudu/127.3.221.125@KRBTEST.COM
I20260811 11:55:06.296011 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.286672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:44917 (local address 127.0.0.1:45802)
0811 11:55:06.286790 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.286804 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 11:55:06.286878 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:55:06.287145 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:55:06.287149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:55:06.287298 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:55:06.287433 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:06.287440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.288466 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.288470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:06.289016 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:06.289025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.289653 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.289656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:06.289738 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.290334 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:55:06.290346 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:55:06.292127 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:55:06.293929 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:06.293932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:06.293934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:06.294127 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:06.294484 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:06.294487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:06.294488 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:06.294589 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:06.295209 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:55:06.295213 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:55:06.295389 (+ 176us) client_negotiation.cc:769] Sending connection context
0811 11:55:06.295597 (+ 208us) client_negotiation.cc:240] Negotiation successful
0811 11:55:06.295821 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260811 11:55:06.296135 8028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.286718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45802 (local address 127.3.221.125:44917)
0811 11:55:06.286829 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.286834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.286847 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:55:06.286941 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.286943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.286984 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.287061 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.287551 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.288301 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.289134 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.289511 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.289627 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.292290 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:06.292294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:06.292297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:06.292324 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:06.293811 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.294278 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.294282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.294284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.294374 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.294715 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.294718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.294719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.294999 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:06.295084 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:06.295698 (+ 614us) server_negotiation.cc:299] Negotiation successful
0811 11:55:06.295926 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":48}
I20260811 11:55:06.297356 8025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:55:06.298524 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.298030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:39311 (local address 127.0.0.1:38068)
0811 11:55:06.298313 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.298434 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
W20260811 11:55:06.299247 7944 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.221.124:39311: connect: Connection refused (error 111)
I20260811 11:55:06.300247 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.299999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:39311 (local address 127.0.0.1:38084)
0811 11:55:06.300136 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.300187 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260811 11:55:06.300442 8025 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.124:39311: Network error: Client connection negotiation failed: client connection to 127.3.221.124:39311: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.124.keytab.
Entry for principal kudu/127.3.221.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.124.keytab.
I20260811 11:55:06.302588 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.124:39311
--webserver_interface=127.3.221.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.124.keytab
--principal=kudu/127.3.221.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:33983,127.3.221.125:44917,127.3.221.124:39311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:55:06.312769 7904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } attempt: 1
I20260811 11:55:06.313766 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.313393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:39311 (local address 127.0.0.1:38090)
0811 11:55:06.313584 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.313663 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260811 11:55:06.314584 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.314360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:39311 (local address 127.0.0.1:38106)
0811 11:55:06.314463 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.314517 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260811 11:55:06.314764 7904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.124:39311: Network error: Client connection negotiation failed: client connection to 127.3.221.124:39311: connect: Connection refused (error 111)
I20260811 11:55:06.349740 8025 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } attempt: 1
I20260811 11:55:06.350991 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.350540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:39311 (local address 127.0.0.1:38118)
0811 11:55:06.350801 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.350887 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260811 11:55:06.351970 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.351682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:39311 (local address 127.0.0.1:38126)
0811 11:55:06.351843 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.351902 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260811 11:55:06.352144 8025 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.124:39311: Network error: Client connection negotiation failed: client connection to 127.3.221.124:39311: connect: Connection refused (error 111)
I20260811 11:55:06.389957 7904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } attempt: 2
I20260811 11:55:06.390980 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.390584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:39311 (local address 127.0.0.1:38142)
0811 11:55:06.390813 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.390880 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260811 11:55:06.391940 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.391659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:39311 (local address 127.0.0.1:38144)
0811 11:55:06.391805 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.391865 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260811 11:55:06.392168 7904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.124:39311: Network error: Client connection negotiation failed: client connection to 127.3.221.124:39311: connect: Connection refused (error 111)
I20260811 11:55:06.420352 8025 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } attempt: 2
I20260811 11:55:06.421307 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.420922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:39311 (local address 127.0.0.1:38150)
0811 11:55:06.421097 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.421173 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260811 11:55:06.422457 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.422190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:39311 (local address 127.0.0.1:38164)
0811 11:55:06.422317 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.422385 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260811 11:55:06.422652 8025 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.124:39311: Network error: Client connection negotiation failed: client connection to 127.3.221.124:39311: connect: Connection refused (error 111)
W20260811 11:55:06.427273 8036 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 11:55:06.427654 8036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:55:06.427691 8036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:55:06.427736 8036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:55:06.432502 8036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:55:06.432581 8036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:55:06.432601 8036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:55:06.432617 8036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:55:06.432632 8036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:55:06.439677 8036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:33983,127.3.221.125:44917,127.3.221.124:39311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.124.keytab
--principal=kudu/127.3.221.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.124:39311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.221.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:55:06.440944 8036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:55:06.442107 8036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 11:55:06.451941 8036 server_base.cc:1034] running on GCE node
W20260811 11:55:06.451861 8053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:55:06.451861 8048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:55:06.451929 8050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:55:06.452885 8036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:55:06.454144 8036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:55:06.455340 8036 hybrid_clock.cc:648] HybridClock initialized: now 1786449306455319 us; error 39 us; skew 500 ppm
Aug 11 11:55:06 dist-test-slave-bndk krb5kdc[7762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:55:06.459157 8036 init.cc:377] Logged in from keytab as kudu/127.3.221.124@KRBTEST.COM (short username kudu)
I20260811 11:55:06.460352 8036 webserver.cc:460] Webserver started at http://127.3.221.124:41571/ using document root <none> and password file <none>
I20260811 11:55:06.460958 8036 fs_manager.cc:359] Metadata directory not provided
I20260811 11:55:06.461031 8036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:55:06.461292 8036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:55:06.463554 8036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-2/data/instance:
uuid: "4e1dddccd6bf48ff9603306555f478cc"
format_stamp: "Formatted at 2026-08-11 11:55:06 on dist-test-slave-bndk"
server_key: "20c845542b8172c1696fc0566c1491d0"
server_key_iv: "cf712dff9daa267e2953f7d30ea03826"
server_key_version: "encryptionkey@0"
I20260811 11:55:06.464176 8036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-2/wal/instance:
uuid: "4e1dddccd6bf48ff9603306555f478cc"
format_stamp: "Formatted at 2026-08-11 11:55:06 on dist-test-slave-bndk"
server_key: "20c845542b8172c1696fc0566c1491d0"
server_key_iv: "cf712dff9daa267e2953f7d30ea03826"
server_key_version: "encryptionkey@0"
I20260811 11:55:06.468410 8036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260811 11:55:06.471549 8061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:55:06.473090 8036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 11:55:06.473268 8036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-2/wal
uuid: "4e1dddccd6bf48ff9603306555f478cc"
format_stamp: "Formatted at 2026-08-11 11:55:06 on dist-test-slave-bndk"
server_key: "20c845542b8172c1696fc0566c1491d0"
server_key_iv: "cf712dff9daa267e2953f7d30ea03826"
server_key_version: "encryptionkey@0"
I20260811 11:55:06.473392 8036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:55:06.483143 8036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:55:06.486572 8036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:55:06.486778 8036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:55:06.491411 7904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } attempt: 3
I20260811 11:55:06.492456 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.491966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:39311 (local address 127.0.0.1:38168)
0811 11:55:06.492162 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.492241 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260811 11:55:06.493201 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.492966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:39311 (local address 127.0.0.1:38174)
0811 11:55:06.493071 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.493127 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260811 11:55:06.493404 7904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.124:39311: Network error: Client connection negotiation failed: client connection to 127.3.221.124:39311: connect: Connection refused (error 111)
I20260811 11:55:06.494657 8036 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.124:39311
I20260811 11:55:06.494935 8141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.124:39311 every 8 connection(s)
I20260811 11:55:06.495662 8036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-2/data/info.pb
I20260811 11:55:06.498662 8142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:55:06.501204 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 8036
I20260811 11:55:06.501375 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786449004114841-3957-0/minicluster-data/master-2/wal/instance
I20260811 11:55:06.501636 3957 external_mini_cluster.cc:1428] Setting key 0ae26f7e01ab58eb4345ea7c463ebbfa
I20260811 11:55:06.502907 8142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33983 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:55:06 dist-test-slave-bndk krb5kdc[7762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.124@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
Aug 11 11:55:06 dist-test-slave-bndk krb5kdc[7762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.124@KRBTEST.COM
I20260811 11:55:06.513809 8025 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } attempt: 3
I20260811 11:55:06.515266 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.504817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58358 (local address 127.3.221.126:33983)
0811 11:55:06.504977 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.504981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.505294 (+ 313us) server_negotiation.cc:407] Connection header received
0811 11:55:06.505564 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.505567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.505625 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.505719 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.506599 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.507088 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.508133 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.508368 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.508454 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.511269 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:06.511274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:06.511277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:06.511311 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:06.512808 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.513441 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.513445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.513447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.513529 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.513899 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.513903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.513905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.514155 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:06.514248 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:06.514958 (+ 710us) server_negotiation.cc:299] Negotiation successful
0811 11:55:06.515099 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 11:55:06.516315 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.502986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38190 (local address 127.3.221.124:39311)
0811 11:55:06.503413 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.503537 (+ 124us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.503583 (+ 46us) server_negotiation.cc:407] Connection header received
0811 11:55:06.504495 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.504516 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.504957 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.505336 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.506169 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.507291 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.508160 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.508575 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.508672 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.511675 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:06.511683 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:06.511697 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:06.511727 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:06.513611 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.514125 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.514131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.514133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.514222 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.514527 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.514531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.514532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.514848 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:06.515041 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:06.515968 (+ 927us) server_negotiation.cc:299] Negotiation successful
0811 11:55:06.516093 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":284,"spinlock_wait_cycles":1408,"thread_start_us":111,"threads_started":1}
I20260811 11:55:06.516460 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.504699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:33983 (local address 127.0.0.1:58358)
0811 11:55:06.505145 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.505196 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 11:55:06.505413 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:55:06.505778 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:55:06.505781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:55:06.506100 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:55:06.506437 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:06.506451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.507192 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.507195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:06.508011 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:06.508021 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.508508 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.508512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:06.508668 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.509307 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:55:06.509325 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:55:06.511010 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:55:06.512939 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:06.512948 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:06.512960 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:06.513341 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:06.513618 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:06.513621 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:06.513623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:06.513775 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:06.514269 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:55:06.514277 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:55:06.514639 (+ 362us) client_negotiation.cc:769] Sending connection context
0811 11:55:06.514882 (+ 243us) client_negotiation.cc:240] Negotiation successful
0811 11:55:06.515247 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":40,"spinlock_wait_cycles":9600,"thread_start_us":178,"threads_started":1}
I20260811 11:55:06.518688 8142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:55:06 dist-test-slave-bndk krb5kdc[7762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.125@KRBTEST.COM for kudu/127.3.221.124@KRBTEST.COM
Aug 11 11:55:06 dist-test-slave-bndk krb5kdc[7762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.124@KRBTEST.COM for kudu/127.3.221.125@KRBTEST.COM
Aug 11 11:55:06 dist-test-slave-bndk krb5kdc[7762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:55:06.525285 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.514306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:39311 (local address 127.0.0.1:38202)
0811 11:55:06.514467 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.514484 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 11:55:06.514567 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:55:06.515193 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:55:06.515196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:55:06.515364 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:55:06.515487 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:06.515494 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.516738 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.516741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:06.517554 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:06.517567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.518217 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.518222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:06.518310 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.518843 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:55:06.518862 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:55:06.520415 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:55:06.523580 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:06.523584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:06.523587 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:06.523844 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:06.524164 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:06.524167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:06.524169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:06.524274 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:06.524670 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:55:06.524673 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:55:06.524785 (+ 112us) client_negotiation.cc:769] Sending connection context
0811 11:55:06.524894 (+ 109us) client_negotiation.cc:240] Negotiation successful
0811 11:55:06.525059 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260811 11:55:06.526182 8155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.514600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38202 (local address 127.3.221.124:39311)
0811 11:55:06.514959 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.514967 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.514978 (+ 11us) server_negotiation.cc:407] Connection header received
0811 11:55:06.515009 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.515011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.515051 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.515600 (+ 549us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.515690 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.516616 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.517696 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.518054 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.518165 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.520539 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:06.520543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:06.520546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:06.520571 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:06.523446 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.523977 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.523980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.523982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.524069 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.524367 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.524370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.524372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.524579 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:06.524659 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:06.524986 (+ 327us) server_negotiation.cc:299] Negotiation successful
0811 11:55:06.525998 (+ 1012us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":136,"threads_started":1}
I20260811 11:55:06.527544 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.519235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:44917 (local address 127.0.0.1:45816)
0811 11:55:06.519351 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.519361 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 11:55:06.519418 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:55:06.519871 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:55:06.519874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:55:06.520058 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:55:06.520183 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:06.520190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.520990 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.520993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:06.521594 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:06.521604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.522028 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.522032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:06.522131 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.522564 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:55:06.522577 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:55:06.523858 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:55:06.525979 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:06.525982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:06.525984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:06.526205 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:06.526563 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:06.526566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:06.526568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:06.526658 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:06.527077 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:55:06.527080 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:55:06.527162 (+ 82us) client_negotiation.cc:769] Sending connection context
0811 11:55:06.527251 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 11:55:06.527396 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260811 11:55:06.527570 8028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.519475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45816 (local address 127.3.221.125:44917)
0811 11:55:06.519589 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.519594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.519607 (+ 13us) server_negotiation.cc:407] Connection header received
0811 11:55:06.519640 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.519643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.519682 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.519752 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.520311 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.520845 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.521708 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.521929 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.522017 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.523992 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:06.523995 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:06.523997 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:06.524018 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:06.525875 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.526341 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.526345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.526347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.526443 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.526750 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.526754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.526756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.526980 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:06.527071 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:06.527337 (+ 266us) server_negotiation.cc:299] Negotiation successful
0811 11:55:06.527442 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260811 11:55:06.528262 8142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:55:06.528996 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.518997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58362 (local address 127.3.221.126:33983)
0811 11:55:06.519591 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.519595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.519605 (+ 10us) server_negotiation.cc:407] Connection header received
0811 11:55:06.519637 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.519639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.519675 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.519747 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.520416 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.521081 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.522212 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.522532 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.522629 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.525588 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:06.525591 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:06.525593 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:06.525619 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:06.527169 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.527650 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.527653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.527655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.527743 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.528069 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.528073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.528074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.528302 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:06.528528 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:06.528720 (+ 192us) server_negotiation.cc:299] Negotiation successful
0811 11:55:06.528847 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525}
I20260811 11:55:06.530059 8025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8215aa3e775b490195f859dfcae15372: Bootstrap starting.
I20260811 11:55:06.533054 8025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8215aa3e775b490195f859dfcae15372: Neither blocks nor log segments found. Creating new log.
Aug 11 11:55:06 dist-test-slave-bndk krb5kdc[7762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.124@KRBTEST.COM for kudu/127.3.221.124@KRBTEST.COM
I20260811 11:55:06.534013 8025 log.cc:824] T 00000000000000000000000000000000 P 8215aa3e775b490195f859dfcae15372: Log is configured to *not* fsync() on all Append() calls
Aug 11 11:55:06 dist-test-slave-bndk krb5kdc[7762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.125@KRBTEST.COM
I20260811 11:55:06.536351 8025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8215aa3e775b490195f859dfcae15372: No bootstrap required, opened a new log
I20260811 11:55:06.538129 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.528971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38208 (local address 127.3.221.124:39311)
0811 11:55:06.529203 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.529207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.529351 (+ 144us) server_negotiation.cc:407] Connection header received
0811 11:55:06.529476 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.529478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.529515 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.529555 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.530024 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.530739 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.531715 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.532079 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.532166 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.534168 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:06.534172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:06.534175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:06.534199 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:06.536066 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.536809 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.536812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.536814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.536916 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.537190 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.537193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.537195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.537448 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:06.537554 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:06.537821 (+ 267us) server_negotiation.cc:299] Negotiation successful
0811 11:55:06.537948 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":156}
I20260811 11:55:06.538559 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.528783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:39311 (local address 127.0.0.1:38208)
0811 11:55:06.529295 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.529308 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:55:06.529391 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:55:06.529670 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:55:06.529672 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:55:06.529849 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:55:06.529924 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:06.529930 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.530834 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.530837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:06.531602 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:06.531614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.532163 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.532166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:06.532272 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.532686 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:55:06.532700 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:55:06.534048 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:55:06.536184 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:06.536187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:06.536189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:06.536684 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:06.536998 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:06.537001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:06.537003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:06.537100 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:06.537560 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:55:06.537564 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:55:06.537660 (+ 96us) client_negotiation.cc:769] Sending connection context
0811 11:55:06.538286 (+ 626us) client_negotiation.cc:240] Negotiation successful
0811 11:55:06.538417 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":3}
I20260811 11:55:06.539551 8028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.530418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45828 (local address 127.3.221.125:44917)
0811 11:55:06.530525 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.530529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.530541 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:55:06.530574 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.530577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.530615 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.530691 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.531372 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.532002 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.532996 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.533325 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.533421 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.536088 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:06.536092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:06.536094 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:06.536119 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:06.537563 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.537934 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.537937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.537939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.538018 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.538439 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.538441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.538442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.538619 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:06.538807 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:06.539281 (+ 474us) server_negotiation.cc:299] Negotiation successful
0811 11:55:06.539404 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 11:55:06.540345 8025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8215aa3e775b490195f859dfcae15372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffc12eb3de74815bfb79cd3d06944de" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33983 } } peers { permanent_uuid: "8215aa3e775b490195f859dfcae15372" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } } peers { permanent_uuid: "4e1dddccd6bf48ff9603306555f478cc" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } }
I20260811 11:55:06.540544 8025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8215aa3e775b490195f859dfcae15372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8215aa3e775b490195f859dfcae15372, State: Initialized, Role: FOLLOWER
I20260811 11:55:06.540853 8025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8215aa3e775b490195f859dfcae15372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffc12eb3de74815bfb79cd3d06944de" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33983 } } peers { permanent_uuid: "8215aa3e775b490195f859dfcae15372" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } } peers { permanent_uuid: "4e1dddccd6bf48ff9603306555f478cc" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } }
I20260811 11:55:06.541379 8159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8215aa3e775b490195f859dfcae15372 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffc12eb3de74815bfb79cd3d06944de" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33983 } } peers { permanent_uuid: "8215aa3e775b490195f859dfcae15372" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } } peers { permanent_uuid: "4e1dddccd6bf48ff9603306555f478cc" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } } }
I20260811 11:55:06.541563 8159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8215aa3e775b490195f859dfcae15372 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:55:06.541548 8142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e1dddccd6bf48ff9603306555f478cc: Bootstrap starting.
I20260811 11:55:06.542158 8025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8215aa3e775b490195f859dfcae15372 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:55:06.543946 8142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e1dddccd6bf48ff9603306555f478cc: Neither blocks nor log segments found. Creating new log.
I20260811 11:55:06.544796 8142 log.cc:824] T 00000000000000000000000000000000 P 4e1dddccd6bf48ff9603306555f478cc: Log is configured to *not* fsync() on all Append() calls
W20260811 11:55:06.547140 8177 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8215aa3e775b490195f859dfcae15372: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:55:06.547274 8177 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:55:06.548570 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:55:06.545951 7915 master_runner.cc:186] Error getting master registration for 127.3.221.126:33983: OK, instance_id { permanent_uuid: "5ffc12eb3de74815bfb79cd3d06944de" instance_seqno: 1786449306014777 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 11:55:06.549253 7915 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 11:55:06.550163 7861 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5ffc12eb3de74815bfb79cd3d06944de"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58362
I20260811 11:55:06.551424 8142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e1dddccd6bf48ff9603306555f478cc: No bootstrap required, opened a new log
I20260811 11:55:06.555892 8142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e1dddccd6bf48ff9603306555f478cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffc12eb3de74815bfb79cd3d06944de" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33983 } } peers { permanent_uuid: "8215aa3e775b490195f859dfcae15372" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } } peers { permanent_uuid: "4e1dddccd6bf48ff9603306555f478cc" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } }
I20260811 11:55:06.556203 8142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e1dddccd6bf48ff9603306555f478cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e1dddccd6bf48ff9603306555f478cc, State: Initialized, Role: FOLLOWER
I20260811 11:55:06.556592 8142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e1dddccd6bf48ff9603306555f478cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffc12eb3de74815bfb79cd3d06944de" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33983 } } peers { permanent_uuid: "8215aa3e775b490195f859dfcae15372" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } } peers { permanent_uuid: "4e1dddccd6bf48ff9603306555f478cc" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } }
I20260811 11:55:06.557191 8184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e1dddccd6bf48ff9603306555f478cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffc12eb3de74815bfb79cd3d06944de" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33983 } } peers { permanent_uuid: "8215aa3e775b490195f859dfcae15372" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } } peers { permanent_uuid: "4e1dddccd6bf48ff9603306555f478cc" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } } }
I20260811 11:55:06.557370 8184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e1dddccd6bf48ff9603306555f478cc [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:55:06.558199 8142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e1dddccd6bf48ff9603306555f478cc [sys.catalog]: configured and running, proceeding with master startup.
W20260811 11:55:06.564138 8201 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e1dddccd6bf48ff9603306555f478cc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:55:06.564250 8201 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:55:06.562585 8036 master_runner.cc:186] Error getting master registration for 127.3.221.126:33983: OK, instance_id { permanent_uuid: "5ffc12eb3de74815bfb79cd3d06944de" instance_seqno: 1786449306014777 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 11:55:06.566522 8036 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 11:55:06.657577 7904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } attempt: 4
Aug 11 11:55:06 dist-test-slave-bndk krb5kdc[7762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449305, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for kudu/127.3.221.124@KRBTEST.COM
I20260811 11:55:06.667990 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.658242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:39311 (local address 127.0.0.1:36922)
0811 11:55:06.658514 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:06.658532 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 11:55:06.658652 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:55:06.659082 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:55:06.659086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:55:06.659321 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:55:06.659443 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:06.659451 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.660480 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.660484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:06.661263 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:06.661276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.661886 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.661890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:06.661984 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.662543 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:55:06.662558 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:55:06.664217 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:55:06.666289 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:06.666293 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:06.666296 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:06.666557 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:06.666856 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:06.666859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:06.666861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:06.666965 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:06.667429 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:55:06.667432 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:55:06.667537 (+ 105us) client_negotiation.cc:769] Sending connection context
0811 11:55:06.667634 (+ 97us) client_negotiation.cc:240] Negotiation successful
0811 11:55:06.667786 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260811 11:55:06.668673 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.658349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36922 (local address 127.3.221.124:39311)
0811 11:55:06.658822 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.658828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.658843 (+ 15us) server_negotiation.cc:407] Connection header received
0811 11:55:06.658889 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.658892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.658938 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.659010 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.659596 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.660331 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.661391 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.661741 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.661851 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.664400 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:06.664405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:06.664408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:06.664439 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:06.666148 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.666669 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.666673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.666675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.666760 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.667063 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.667066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.667068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.667296 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:06.667383 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:06.668386 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0811 11:55:06.668520 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386}
I20260811 11:55:06.672715 7904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de: Bootstrap starting.
I20260811 11:55:06.675825 7904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de: Neither blocks nor log segments found. Creating new log.
I20260811 11:55:06.676982 7904 log.cc:824] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de: Log is configured to *not* fsync() on all Append() calls
I20260811 11:55:06.679644 7904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de: No bootstrap required, opened a new log
I20260811 11:55:06.682961 7904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffc12eb3de74815bfb79cd3d06944de" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33983 } } peers { permanent_uuid: "8215aa3e775b490195f859dfcae15372" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } } peers { permanent_uuid: "4e1dddccd6bf48ff9603306555f478cc" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } }
I20260811 11:55:06.683300 7904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ffc12eb3de74815bfb79cd3d06944de, State: Initialized, Role: FOLLOWER
I20260811 11:55:06.683701 7904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffc12eb3de74815bfb79cd3d06944de" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33983 } } peers { permanent_uuid: "8215aa3e775b490195f859dfcae15372" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } } peers { permanent_uuid: "4e1dddccd6bf48ff9603306555f478cc" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } }
I20260811 11:55:06.685351 7904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:55:06.685454 7861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260811 11:55:06.685621 7861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:55:06.687125 7861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffc12eb3de74815bfb79cd3d06944de" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33983 } } peers { permanent_uuid: "8215aa3e775b490195f859dfcae15372" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } } peers { permanent_uuid: "4e1dddccd6bf48ff9603306555f478cc" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } }
I20260811 11:55:06.688738 7983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5ffc12eb3de74815bfb79cd3d06944de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8215aa3e775b490195f859dfcae15372"
I20260811 11:55:06.689025 7983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8215aa3e775b490195f859dfcae15372 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:55:06.690280 7861 leader_election.cc:288] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de [CANDIDATE]: Term 1 election: Requested vote from peers 8215aa3e775b490195f859dfcae15372 (127.3.221.125:44917), 4e1dddccd6bf48ff9603306555f478cc (127.3.221.124:39311)
I20260811 11:55:06.690667 7983 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8215aa3e775b490195f859dfcae15372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ffc12eb3de74815bfb79cd3d06944de in term 1.
I20260811 11:55:06.691203 8104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5ffc12eb3de74815bfb79cd3d06944de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4e1dddccd6bf48ff9603306555f478cc"
I20260811 11:55:06.691489 8104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e1dddccd6bf48ff9603306555f478cc [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:55:06.693804 7821 leader_election.cc:302] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ffc12eb3de74815bfb79cd3d06944de, 8215aa3e775b490195f859dfcae15372; no voters:
I20260811 11:55:06.695154 8104 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4e1dddccd6bf48ff9603306555f478cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ffc12eb3de74815bfb79cd3d06944de in term 1.
I20260811 11:55:06.688083 8205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffc12eb3de74815bfb79cd3d06944de" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33983 } } peers { permanent_uuid: "8215aa3e775b490195f859dfcae15372" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } } peers { permanent_uuid: "4e1dddccd6bf48ff9603306555f478cc" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } } }
I20260811 11:55:06.695600 8205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:55:06.699411 7784 master_runner.cc:263] No leader master found from master 5ffc12eb3de74815bfb79cd3d06944de
I20260811 11:55:06.699486 7784 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260811 11:55:06.697106 8223 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:55:06.699544 8223 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:55:06.697355 8233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:55:06.699678 8233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de [term 1 LEADER]: Becoming Leader. State: Replica: 5ffc12eb3de74815bfb79cd3d06944de, State: Running, Role: LEADER
I20260811 11:55:06.700031 8233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffc12eb3de74815bfb79cd3d06944de" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33983 } } peers { permanent_uuid: "8215aa3e775b490195f859dfcae15372" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } } peers { permanent_uuid: "4e1dddccd6bf48ff9603306555f478cc" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } }
I20260811 11:55:06.702587 8205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ffc12eb3de74815bfb79cd3d06944de. Latest consensus state: current_term: 1 leader_uuid: "5ffc12eb3de74815bfb79cd3d06944de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffc12eb3de74815bfb79cd3d06944de" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33983 } } peers { permanent_uuid: "8215aa3e775b490195f859dfcae15372" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } } peers { permanent_uuid: "4e1dddccd6bf48ff9603306555f478cc" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } } }
I20260811 11:55:06.702734 8205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de [sys.catalog]: This master's current role is: LEADER
I20260811 11:55:06.703209 8239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:55:06.704210 8239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:55:06.710232 7983 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8215aa3e775b490195f859dfcae15372 [term 1 FOLLOWER]: Refusing update from remote peer 5ffc12eb3de74815bfb79cd3d06944de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:55:06.710227 8104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e1dddccd6bf48ff9603306555f478cc [term 1 FOLLOWER]: Refusing update from remote peer 5ffc12eb3de74815bfb79cd3d06944de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:55:06.710966 8205 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e1dddccd6bf48ff9603306555f478cc" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:55:06.711562 8233 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de [LEADER]: Connected to new peer: Peer: permanent_uuid: "8215aa3e775b490195f859dfcae15372" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:55:06.716481 8184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e1dddccd6bf48ff9603306555f478cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ffc12eb3de74815bfb79cd3d06944de. Latest consensus state: current_term: 1 leader_uuid: "5ffc12eb3de74815bfb79cd3d06944de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffc12eb3de74815bfb79cd3d06944de" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33983 } } peers { permanent_uuid: "8215aa3e775b490195f859dfcae15372" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } } peers { permanent_uuid: "4e1dddccd6bf48ff9603306555f478cc" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } } }
I20260811 11:55:06.716629 8184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e1dddccd6bf48ff9603306555f478cc [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:55:06.718484 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.698666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36938 (local address 127.3.221.124:39311)
0811 11:55:06.698777 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.698781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.709504 (+ 10723us) server_negotiation.cc:407] Connection header received
0811 11:55:06.709690 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.709694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.709750 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.709805 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.710804 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.711494 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.712516 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.712823 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.712899 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.714746 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:06.714750 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:06.714752 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:06.714783 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:06.716482 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.717000 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.717003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.717005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.717099 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.717461 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.717464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.717466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.717707 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:06.717811 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:06.718193 (+ 382us) server_negotiation.cc:299] Negotiation successful
0811 11:55:06.718319 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 11:55:06.721028 8159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8215aa3e775b490195f859dfcae15372 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ffc12eb3de74815bfb79cd3d06944de. Latest consensus state: current_term: 1 leader_uuid: "5ffc12eb3de74815bfb79cd3d06944de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffc12eb3de74815bfb79cd3d06944de" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33983 } } peers { permanent_uuid: "8215aa3e775b490195f859dfcae15372" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } } peers { permanent_uuid: "4e1dddccd6bf48ff9603306555f478cc" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } } }
I20260811 11:55:06.721191 8159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8215aa3e775b490195f859dfcae15372 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:55:06.732002 8242 mvcc.cc:205] Tried to move back new op lower bound from 7317296360276422656 to 7317296360247480320. Current Snapshot: MvccSnapshot[applied={T|T < 7317296360276422656}]
I20260811 11:55:06.732925 8205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ffc12eb3de74815bfb79cd3d06944de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffc12eb3de74815bfb79cd3d06944de" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33983 } } peers { permanent_uuid: "8215aa3e775b490195f859dfcae15372" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } } peers { permanent_uuid: "4e1dddccd6bf48ff9603306555f478cc" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } } }
I20260811 11:55:06.733073 8205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de [sys.catalog]: This master's current role is: LEADER
I20260811 11:55:06.733708 8159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8215aa3e775b490195f859dfcae15372 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ffc12eb3de74815bfb79cd3d06944de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffc12eb3de74815bfb79cd3d06944de" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33983 } } peers { permanent_uuid: "8215aa3e775b490195f859dfcae15372" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } } peers { permanent_uuid: "4e1dddccd6bf48ff9603306555f478cc" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } } }
I20260811 11:55:06.733850 8159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8215aa3e775b490195f859dfcae15372 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:55:06.734087 8184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e1dddccd6bf48ff9603306555f478cc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ffc12eb3de74815bfb79cd3d06944de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffc12eb3de74815bfb79cd3d06944de" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33983 } } peers { permanent_uuid: "8215aa3e775b490195f859dfcae15372" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } } peers { permanent_uuid: "4e1dddccd6bf48ff9603306555f478cc" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } } }
I20260811 11:55:06.734215 8184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e1dddccd6bf48ff9603306555f478cc [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:55:06.734421 8245 mvcc.cc:205] Tried to move back new op lower bound from 7317296360276422656 to 7317296360247480320. Current Snapshot: MvccSnapshot[applied={T|T < 7317296360276422656}]
I20260811 11:55:06.736289 8239 catalog_manager.cc:1347] Generated new cluster ID: 3650e6a2f66d4079943bdb938b21a355
I20260811 11:55:06.736362 8239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:55:06.737910 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.699090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39156 (local address 127.3.221.126:33983)
0811 11:55:06.699200 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.699205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.701725 (+ 2520us) server_negotiation.cc:407] Connection header received
0811 11:55:06.703219 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.703223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.703276 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.703329 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.706433 (+ 3104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.707107 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.728261 (+ 21154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.728686 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.728782 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.733508 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:06.733512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:06.733515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:06.733550 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:06.735489 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.736438 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.736442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.736444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.736536 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.736815 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.736819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.736820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.737076 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:06.737247 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:06.737623 (+ 376us) server_negotiation.cc:299] Negotiation successful
0811 11:55:06.737752 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 11:55:06.736279 8233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ffc12eb3de74815bfb79cd3d06944de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffc12eb3de74815bfb79cd3d06944de" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33983 } } peers { permanent_uuid: "8215aa3e775b490195f859dfcae15372" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 44917 } } peers { permanent_uuid: "4e1dddccd6bf48ff9603306555f478cc" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 39311 } } }
I20260811 11:55:06.741463 8233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de [sys.catalog]: This master's current role is: LEADER
I20260811 11:55:06.742059 8243 mvcc.cc:205] Tried to move back new op lower bound from 7317296360276422656 to 7317296360247480320. Current Snapshot: MvccSnapshot[applied={T|T < 7317296360276422656}]
I20260811 11:55:06.746758 8028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:06.698017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60162 (local address 127.3.221.125:44917)
0811 11:55:06.698137 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.698141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.700390 (+ 2249us) server_negotiation.cc:407] Connection header received
0811 11:55:06.700587 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.700590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.700641 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.700693 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.704407 (+ 3714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.705212 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.708296 (+ 3084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:06.708634 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:06.708725 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:06.725467 (+ 16742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:06.725472 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:06.725475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:06.725517 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:06.727539 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.741993 (+ 14454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.741997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.741999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.742120 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:06.745585 (+ 3465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:06.745589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:06.745591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:06.745861 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:06.745998 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:06.746421 (+ 423us) server_negotiation.cc:299] Negotiation successful
0811 11:55:06.746588 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260811 11:55:06.763749 8239 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:55:06.765161 8239 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:55:06.773273 8239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ffc12eb3de74815bfb79cd3d06944de: Generated new TSK 0
I20260811 11:55:06.774240 8239 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 11:55:06.799376 7911 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:06.797816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39170 (local address 127.3.221.126:33983)
0811 11:55:06.797965 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.797970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.798187 (+ 217us) server_negotiation.cc:407] Connection header received
0811 11:55:06.798368 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.798372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.798445 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.798506 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.799213 (+ 707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39170: BlockingRecv error: recv got EOF from 127.0.0.1:39170 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260811 11:55:06.800179 8028 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:06.798379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60172 (local address 127.3.221.125:44917)
0811 11:55:06.798587 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.798591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.799193 (+ 602us) server_negotiation.cc:407] Connection header received
0811 11:55:06.799363 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.799367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.799418 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.799473 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.800056 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60172: BlockingRecv error: recv got EOF from 127.0.0.1:60172 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
W20260811 11:55:06.800863 8150 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:06.799202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36946 (local address 127.3.221.124:39311)
0811 11:55:06.799630 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.799635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.799905 (+ 270us) server_negotiation.cc:407] Connection header received
0811 11:55:06.800103 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.800107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.800157 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.800211 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.800742 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36946: BlockingRecv error: recv got EOF from 127.0.0.1:36946 (error 108)
Metrics: {"server-negotiator.queue_time_us":290}
W20260811 11:55:06.803900 8028 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:06.802787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60180 (local address 127.3.221.125:44917)
0811 11:55:06.802909 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.802913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.803102 (+ 189us) server_negotiation.cc:407] Connection header received
0811 11:55:06.803253 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.803256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.803324 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.803400 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.803798 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60180: BlockingRecv error: recv got EOF from 127.0.0.1:60180 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260811 11:55:06.808179 8028 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:06.807209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60188 (local address 127.3.221.125:44917)
0811 11:55:06.807315 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.807319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.807338 (+ 19us) server_negotiation.cc:407] Connection header received
0811 11:55:06.807381 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.807383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.807505 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.807587 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.808073 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60188: BlockingRecv error: recv got EOF from 127.0.0.1:60188 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260811 11:55:06.813686 8028 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:06.812501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60190 (local address 127.3.221.125:44917)
0811 11:55:06.812699 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.812704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.812783 (+ 79us) server_negotiation.cc:407] Connection header received
0811 11:55:06.812920 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.812923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.813002 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.813074 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.813572 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60190: BlockingRecv error: recv got EOF from 127.0.0.1:60190 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260811 11:55:06.820830 8028 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:06.819621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60204 (local address 127.3.221.125:44917)
0811 11:55:06.819780 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.819784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.819874 (+ 90us) server_negotiation.cc:407] Connection header received
0811 11:55:06.820070 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.820073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.820124 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.820201 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.820713 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60204: BlockingRecv error: recv got EOF from 127.0.0.1:60204 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260811 11:55:06.832311 8028 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:06.831141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60208 (local address 127.3.221.125:44917)
0811 11:55:06.831291 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.831295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.831439 (+ 144us) server_negotiation.cc:407] Connection header received
0811 11:55:06.831572 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.831575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.831631 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.831727 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.832183 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60208: BlockingRecv error: recv got EOF from 127.0.0.1:60208 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260811 11:55:06.852007 8028 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:06.850723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.3.221.125:44917)
0811 11:55:06.850908 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.850914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.851018 (+ 104us) server_negotiation.cc:407] Connection header received
0811 11:55:06.851156 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.851161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.851235 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.851356 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.851881 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60220: BlockingRecv error: recv got EOF from 127.0.0.1:60220 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260811 11:55:06.889109 8028 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:06.887319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60236 (local address 127.3.221.125:44917)
0811 11:55:06.887496 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.887500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.887698 (+ 198us) server_negotiation.cc:407] Connection header received
0811 11:55:06.887970 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.887973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.888051 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.888168 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.888960 (+ 792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60236: BlockingRecv error: recv got EOF from 127.0.0.1:60236 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260811 11:55:06.957118 8028 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:06.955965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.3.221.125:44917)
0811 11:55:06.956135 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:06.956139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:06.956325 (+ 186us) server_negotiation.cc:407] Connection header received
0811 11:55:06.956428 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:06.956431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:06.956485 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:06.956562 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:06.957015 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60242: BlockingRecv error: recv got EOF from 127.0.0.1:60242 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260811 11:55:07.088914 8028 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:07.087778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60244 (local address 127.3.221.125:44917)
0811 11:55:07.087945 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:07.087949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:07.088020 (+ 71us) server_negotiation.cc:407] Connection header received
0811 11:55:07.088192 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:07.088195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:07.088257 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:07.088361 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:07.088792 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60244: BlockingRecv error: recv got EOF from 127.0.0.1:60244 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260811 11:55:07.348344 8028 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:07.347376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60258 (local address 127.3.221.125:44917)
0811 11:55:07.347553 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:07.347557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:07.347574 (+ 17us) server_negotiation.cc:407] Connection header received
0811 11:55:07.347631 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:07.347634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:07.347682 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:07.347767 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:07.348231 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60258: BlockingRecv error: recv got EOF from 127.0.0.1:60258 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260811 11:55:07.548766 8177 catalog_manager.cc:1259] Loaded cluster ID: 3650e6a2f66d4079943bdb938b21a355
I20260811 11:55:07.548867 8177 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8215aa3e775b490195f859dfcae15372: loading cluster ID for follower catalog manager: success
I20260811 11:55:07.550611 8177 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8215aa3e775b490195f859dfcae15372: acquiring CA information for follower catalog manager: success
I20260811 11:55:07.552004 8177 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8215aa3e775b490195f859dfcae15372: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 11:55:07.565690 8201 catalog_manager.cc:1259] Loaded cluster ID: 3650e6a2f66d4079943bdb938b21a355
I20260811 11:55:07.565784 8201 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4e1dddccd6bf48ff9603306555f478cc: loading cluster ID for follower catalog manager: success
I20260811 11:55:07.567675 8201 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4e1dddccd6bf48ff9603306555f478cc: acquiring CA information for follower catalog manager: success
I20260811 11:55:07.568977 8201 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4e1dddccd6bf48ff9603306555f478cc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 11:55:07.866654 8368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:07.862603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60260 (local address 127.3.221.125:44917)
0811 11:55:07.862861 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:07.862864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:07.862884 (+ 20us) server_negotiation.cc:407] Connection header received
0811 11:55:07.862948 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:07.862950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:07.863080 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:07.863193 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:55:07.863484 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:07.864042 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:07.865078 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:07.865376 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:07.865487 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:07.865790 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:55:07.866247 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:55:07.866337 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:07.866462 (+ 125us) server_negotiation.cc:299] Negotiation successful
0811 11:55:07.866519 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":76,"threads_started":1}
I20260811 11:55:07.874040 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:07.869998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36952 (local address 127.3.221.124:39311)
0811 11:55:07.870235 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:07.870240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:07.870255 (+ 15us) server_negotiation.cc:407] Connection header received
0811 11:55:07.870365 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:07.870369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:07.870495 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:07.870603 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:55:07.870819 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:07.871359 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:07.872198 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:07.872429 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:07.872517 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:07.872810 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:55:07.873405 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:55:07.873525 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:07.873670 (+ 145us) server_negotiation.cc:299] Negotiation successful
0811 11:55:07.873738 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":64,"threads_started":1}
I20260811 11:55:07.881677 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:07.878072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36960 (local address 127.3.221.124:39311)
0811 11:55:07.878180 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:07.878184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:07.878341 (+ 157us) server_negotiation.cc:407] Connection header received
0811 11:55:07.878444 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:07.878447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:07.878492 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:07.878553 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:55:07.878878 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:07.879713 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:07.880588 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:07.880811 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:07.880900 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:07.881150 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:55:07.881259 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:55:07.881324 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:07.881475 (+ 151us) server_negotiation.cc:299] Negotiation successful
0811 11:55:07.881535 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 11:55:07.881867 8368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:07.877478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60274 (local address 127.3.221.125:44917)
0811 11:55:07.877636 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:07.877641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:07.877909 (+ 268us) server_negotiation.cc:407] Connection header received
0811 11:55:07.878132 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:07.878136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:07.878187 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:07.878233 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:55:07.878670 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:07.879592 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:07.880682 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:07.881004 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:07.881073 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:07.881440 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:55:07.881511 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:55:07.881570 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:07.881661 (+ 91us) server_negotiation.cc:299] Negotiation successful
0811 11:55:07.881716 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 11:55:07.882246 8384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:07.877478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39178 (local address 127.3.221.126:33983)
0811 11:55:07.877740 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:07.877744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:07.877926 (+ 182us) server_negotiation.cc:407] Connection header received
0811 11:55:07.878176 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:07.878180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:07.878350 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:07.878444 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:55:07.878847 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:07.879372 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:07.880542 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:07.880885 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:07.880955 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:07.881404 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:55:07.881786 (+ 382us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:55:07.881872 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:07.882027 (+ 155us) server_negotiation.cc:299] Negotiation successful
0811 11:55:07.882095 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":105,"threads_started":1}
I20260811 11:55:07.886746 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 7784
I20260811 11:55:07.906553 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 7915
I20260811 11:55:07.917145 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 8036
2026-08-11T11:55:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3439 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 11:55:07 dist-test-slave-bndk krb5kdc[8393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 11:55:07 dist-test-slave-bndk krb5kdc[8393](info): set up 2 sockets
Aug 11 11:55:07 dist-test-slave-bndk krb5kdc[8393](info): commencing operation
krb5kdc: starting...
W20260811 11:55:09.133989 3957 mini_kdc.cc:121] Time spent starting KDC: real 1.182s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/test-user.keytab.
Aug 11 11:55:09 dist-test-slave-bndk krb5kdc[8393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T11:55:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T11:55:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
Entry for principal kudu/127.3.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab.
I20260811 11:55:09.276961 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:33247
--webserver_interface=127.3.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:41777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:33247,127.3.221.125:36523,127.3.221.124:46491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 11:55:09.407531 8414 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 11:55:09.407907 8414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:55:09.407932 8414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:55:09.407986 8414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:55:09.412477 8414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:55:09.412576 8414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:55:09.412596 8414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:55:09.412626 8414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:55:09.412639 8414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:55:09.420985 8414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:41777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:33247,127.3.221.125:36523,127.3.221.124:46491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.126.keytab
--principal=kudu/127.3.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.126:33247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:55:09.422859 8414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:55:09.424737 8414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:55:09.434998 8431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:55:09.435027 8427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:55:09.434993 8426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:55:09.435587 8414 server_base.cc:1034] running on GCE node
I20260811 11:55:09.436281 8414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:55:09.437633 8414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:55:09.439019 8414 hybrid_clock.cc:648] HybridClock initialized: now 1786449309438985 us; error 73 us; skew 500 ppm
Aug 11 11:55:09 dist-test-slave-bndk krb5kdc[8393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:55:09.446434 8414 init.cc:377] Logged in from keytab as kudu/127.3.221.126@KRBTEST.COM (short username kudu)
I20260811 11:55:09.447608 8414 webserver.cc:460] Webserver started at http://127.3.221.126:36103/ using document root <none> and password file <none>
I20260811 11:55:09.448225 8414 fs_manager.cc:359] Metadata directory not provided
I20260811 11:55:09.448293 8414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:55:09.448513 8414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:55:09.451081 8414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-0/data/instance:
uuid: "d2c60b746f6c4d199836aac0f09c357d"
format_stamp: "Formatted at 2026-08-11 11:55:09 on dist-test-slave-bndk"
server_key: "e6810ce4bef3944b61917d78ff0c9a84"
server_key_iv: "3781d83eaa38041903232a83bfd788b5"
server_key_version: "encryptionkey@0"
I20260811 11:55:09.451788 8414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-0/wal/instance:
uuid: "d2c60b746f6c4d199836aac0f09c357d"
format_stamp: "Formatted at 2026-08-11 11:55:09 on dist-test-slave-bndk"
server_key: "e6810ce4bef3944b61917d78ff0c9a84"
server_key_iv: "3781d83eaa38041903232a83bfd788b5"
server_key_version: "encryptionkey@0"
I20260811 11:55:09.456704 8414 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 11:55:09.460641 8440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:55:09.462575 8414 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 11:55:09.462766 8414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-0/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-0/wal
uuid: "d2c60b746f6c4d199836aac0f09c357d"
format_stamp: "Formatted at 2026-08-11 11:55:09 on dist-test-slave-bndk"
server_key: "e6810ce4bef3944b61917d78ff0c9a84"
server_key_iv: "3781d83eaa38041903232a83bfd788b5"
server_key_version: "encryptionkey@0"
I20260811 11:55:09.462916 8414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:55:09.492760 8414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:55:09.500360 8414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:55:09.500777 8414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:55:09.510867 8414 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.126:33247
I20260811 11:55:09.511404 8529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.126:33247 every 8 connection(s)
I20260811 11:55:09.512699 8414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-0/data/info.pb
I20260811 11:55:09.516260 8530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:55:09.518477 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 8414
I20260811 11:55:09.518625 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-0/wal/instance
I20260811 11:55:09.519089 3957 external_mini_cluster.cc:1428] Setting key ccab26ce94d9be614bbb5752d526b0ae
I20260811 11:55:09.521669 8530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33247 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.221.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.125@KRBTEST.COM" created.
Aug 11 11:55:09 dist-test-slave-bndk krb5kdc[8393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:55:09.540836 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.524308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:33247 (local address 127.0.0.1:54002)
0811 11:55:09.524900 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:09.524957 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 11:55:09.525945 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:55:09.526690 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:55:09.526695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:55:09.527175 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:55:09.527443 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:09.527457 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:09.528693 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:09.528696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:09.529553 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:09.529564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:09.530248 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:09.530252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:09.530368 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:09.530993 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:55:09.531011 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:55:09.532773 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:55:09.537021 (+ 4248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:09.537032 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:09.537044 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:09.537405 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:09.538019 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:09.538022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:09.538023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:09.538169 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:09.539066 (+ 897us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:55:09.539072 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:55:09.539274 (+ 202us) client_negotiation.cc:769] Sending connection context
0811 11:55:09.539464 (+ 190us) client_negotiation.cc:240] Negotiation successful
0811 11:55:09.539817 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":155,"threads_started":1}
I20260811 11:55:09.540840 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.524307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54002 (local address 127.3.221.126:33247)
0811 11:55:09.524907 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:09.524941 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:09.525074 (+ 133us) server_negotiation.cc:407] Connection header received
0811 11:55:09.526275 (+ 1201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:09.526280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:09.526494 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:09.526672 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:09.527617 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:09.528566 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:09.529668 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:09.530074 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:09.530237 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:09.533128 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:09.533139 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:09.533160 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:09.533210 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:09.536758 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:09.537708 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:09.537716 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:09.537717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:09.537885 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:09.538282 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:09.538286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:09.538288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:09.538916 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:09.539064 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:09.539604 (+ 540us) server_negotiation.cc:299] Negotiation successful
0811 11:55:09.539956 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":489,"spinlock_wait_cycles":37248,"thread_start_us":182,"threads_started":1}
I20260811 11:55:09.544294 8530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:55:09.545270 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.544906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:36523 (local address 127.0.0.1:42678)
0811 11:55:09.545072 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:09.545151 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:36523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260811 11:55:09.545548 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.221.125:36523: connect: Connection refused (error 111)
I20260811 11:55:09.546567 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.546336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:36523 (local address 127.0.0.1:42694)
0811 11:55:09.546449 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:09.546499 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:36523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260811 11:55:09.546700 8530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.125:36523: Network error: Client connection negotiation failed: client connection to 127.3.221.125:36523: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.125.keytab.
Entry for principal kudu/127.3.221.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.125.keytab.
I20260811 11:55:09.553702 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.125:36523
--webserver_interface=127.3.221.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.125.keytab
--principal=kudu/127.3.221.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:41777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:33247,127.3.221.125:36523,127.3.221.124:46491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:55:09.596025 8530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } attempt: 1
I20260811 11:55:09.597472 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.596955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:36523 (local address 127.0.0.1:42700)
0811 11:55:09.597209 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:09.597358 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:36523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260811 11:55:09.598670 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.598357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:36523 (local address 127.0.0.1:42710)
0811 11:55:09.598504 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:09.598581 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:36523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260811 11:55:09.599174 8530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.125:36523: Network error: Client connection negotiation failed: client connection to 127.3.221.125:36523: connect: Connection refused (error 111)
I20260811 11:55:09.667449 8530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } attempt: 2
I20260811 11:55:09.668354 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.668001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:36523 (local address 127.0.0.1:42722)
0811 11:55:09.668177 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:09.668255 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:36523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260811 11:55:09.669391 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.669065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:36523 (local address 127.0.0.1:42728)
0811 11:55:09.669186 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:09.669303 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:36523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260811 11:55:09.669552 8530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.125:36523: Network error: Client connection negotiation failed: client connection to 127.3.221.125:36523: connect: Connection refused (error 111)
W20260811 11:55:09.692970 8540 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 11:55:09.693461 8540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:55:09.693562 8540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:55:09.693629 8540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:55:09.699913 8540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:55:09.700078 8540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:55:09.700121 8540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:55:09.700158 8540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:55:09.700191 8540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:55:09.707644 8540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:41777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:33247,127.3.221.125:36523,127.3.221.124:46491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.125.keytab
--principal=kudu/127.3.221.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.125:36523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.221.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:55:09.708957 8540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:55:09.710186 8540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:55:09.720360 8551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:55:09.720443 8556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:55:09.720484 8540 server_base.cc:1034] running on GCE node
W20260811 11:55:09.720360 8553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:55:09.721361 8540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:55:09.722618 8540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:55:09.723829 8540 hybrid_clock.cc:648] HybridClock initialized: now 1786449309723801 us; error 48 us; skew 500 ppm
Aug 11 11:55:09 dist-test-slave-bndk krb5kdc[8393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:55:09.727028 8540 init.cc:377] Logged in from keytab as kudu/127.3.221.125@KRBTEST.COM (short username kudu)
I20260811 11:55:09.728107 8540 webserver.cc:460] Webserver started at http://127.3.221.125:35715/ using document root <none> and password file <none>
I20260811 11:55:09.728737 8540 fs_manager.cc:359] Metadata directory not provided
I20260811 11:55:09.728822 8540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:55:09.729058 8540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:55:09.731395 8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-1/data/instance:
uuid: "8bfdc03be7484a3db1490e429309cb11"
format_stamp: "Formatted at 2026-08-11 11:55:09 on dist-test-slave-bndk"
server_key: "2235bed09702a48a38b8794acd52f9e1"
server_key_iv: "2fe431645ba1a778d258fd4c0aef15d2"
server_key_version: "encryptionkey@0"
I20260811 11:55:09.732013 8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-1/wal/instance:
uuid: "8bfdc03be7484a3db1490e429309cb11"
format_stamp: "Formatted at 2026-08-11 11:55:09 on dist-test-slave-bndk"
server_key: "2235bed09702a48a38b8794acd52f9e1"
server_key_iv: "2fe431645ba1a778d258fd4c0aef15d2"
server_key_version: "encryptionkey@0"
I20260811 11:55:09.738195 8540 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260811 11:55:09.741189 8566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:55:09.742842 8540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 11:55:09.743001 8540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-1/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-1/wal
uuid: "8bfdc03be7484a3db1490e429309cb11"
format_stamp: "Formatted at 2026-08-11 11:55:09 on dist-test-slave-bndk"
server_key: "2235bed09702a48a38b8794acd52f9e1"
server_key_iv: "2fe431645ba1a778d258fd4c0aef15d2"
server_key_version: "encryptionkey@0"
I20260811 11:55:09.743162 8540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:55:09.760738 8530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } attempt: 3
I20260811 11:55:09.761807 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.761478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:36523 (local address 127.0.0.1:42744)
0811 11:55:09.761638 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:09.761715 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:36523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260811 11:55:09.762605 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.762395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:36523 (local address 127.0.0.1:42746)
0811 11:55:09.762489 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:09.762542 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.125:36523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260811 11:55:09.762764 8530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.125:36523: Network error: Client connection negotiation failed: client connection to 127.3.221.125:36523: connect: Connection refused (error 111)
I20260811 11:55:09.765843 8540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:55:09.769649 8540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:55:09.769841 8540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:55:09.778323 8540 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.125:36523
I20260811 11:55:09.778626 8659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.125:36523 every 8 connection(s)
I20260811 11:55:09.779510 8540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-1/data/info.pb
I20260811 11:55:09.782752 8660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:55:09.784046 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 8540
I20260811 11:55:09.784176 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-1/wal/instance
I20260811 11:55:09.784463 3957 external_mini_cluster.cc:1428] Setting key 081f94fabd288ea012925360e778d3cb
I20260811 11:55:09.787281 8660 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33247 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.221.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.221.124@KRBTEST.COM" created.
Aug 11 11:55:09 dist-test-slave-bndk krb5kdc[8393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.125@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:55:09.806762 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.789411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54018 (local address 127.3.221.126:33247)
0811 11:55:09.789578 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:09.789583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:09.790566 (+ 983us) server_negotiation.cc:407] Connection header received
0811 11:55:09.792216 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:09.792221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:09.792292 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:09.792555 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:09.794643 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:09.795232 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:09.798319 (+ 3087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:09.798589 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:09.798695 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:09.802691 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:09.802694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:09.802697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:09.802738 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:09.804186 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:09.804826 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:09.804829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:09.804830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:09.804914 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:09.805324 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:09.805328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:09.805329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:09.805543 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:09.805639 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:09.806487 (+ 848us) server_negotiation.cc:299] Negotiation successful
0811 11:55:09.806627 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 11:55:09.808039 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.789801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:33247 (local address 127.0.0.1:54018)
0811 11:55:09.790333 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:09.790397 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0811 11:55:09.791932 (+ 1535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:55:09.792747 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:55:09.792754 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:55:09.793134 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:55:09.794354 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:09.794384 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:09.795590 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:09.795598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:09.798076 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:09.798099 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:09.798808 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:09.798815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:09.799009 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:09.799986 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:55:09.800019 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:55:09.802438 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:55:09.804345 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:09.804354 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:09.804376 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:09.804706 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:09.805007 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:09.805011 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:09.805013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:09.805185 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:09.805656 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:55:09.805666 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:55:09.806083 (+ 417us) client_negotiation.cc:769] Sending connection context
0811 11:55:09.806350 (+ 267us) client_negotiation.cc:240] Negotiation successful
0811 11:55:09.806790 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":133,"threads_started":1}
I20260811 11:55:09.809754 8660 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:55:09 dist-test-slave-bndk krb5kdc[8393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.125@KRBTEST.COM for kudu/127.3.221.125@KRBTEST.COM
I20260811 11:55:09.824290 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.810339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:36523 (local address 127.0.0.1:42750)
0811 11:55:09.810487 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:09.810500 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 11:55:09.810580 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:55:09.811322 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:55:09.811324 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:55:09.811601 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:55:09.811714 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:09.811720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:09.813676 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:09.813680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:09.814788 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:09.814805 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:09.816884 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:09.816887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:09.816990 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:09.817560 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:55:09.817573 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:55:09.819216 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:55:09.821949 (+ 2733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:09.821952 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:09.821954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:09.822186 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:09.822997 (+ 811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:09.823000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:09.823001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:09.823108 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:09.823797 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:55:09.823801 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:55:09.823899 (+ 98us) client_negotiation.cc:769] Sending connection context
0811 11:55:09.823998 (+ 99us) client_negotiation.cc:240] Negotiation successful
0811 11:55:09.824136 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260811 11:55:09.824496 8669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.810473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42750 (local address 127.3.221.125:36523)
0811 11:55:09.810809 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:09.810819 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:09.810841 (+ 22us) server_negotiation.cc:407] Connection header received
0811 11:55:09.810889 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:09.810895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:09.811130 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:09.811265 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:09.811874 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:09.812998 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:09.815923 (+ 2925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:09.816662 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:09.816793 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:09.819406 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:09.819420 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:09.819437 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:09.819463 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:09.821783 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:09.822556 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:09.822567 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:09.822568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:09.822801 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:09.823251 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:09.823255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:09.823256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:09.823652 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:09.823802 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:09.824126 (+ 324us) server_negotiation.cc:299] Negotiation successful
0811 11:55:09.824272 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":144,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.221.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.124.keytab.
Entry for principal kudu/127.3.221.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.124.keytab.
I20260811 11:55:09.826682 3957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt5cui_/build/debug/bin/kudu
/tmp/dist-test-taskt5cui_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.124:46491
--webserver_interface=127.3.221.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.124.keytab
--principal=kudu/127.3.221.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.221.84:41777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.221.126:33247,127.3.221.125:36523,127.3.221.124:46491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 11:55:09.826711 8660 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:55:09.827486 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.827199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:46491 (local address 127.0.0.1:47514)
0811 11:55:09.827328 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:09.827408 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:46491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260811 11:55:09.827699 8571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.221.124:46491: connect: Connection refused (error 111)
I20260811 11:55:09.828713 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.828466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:46491 (local address 127.0.0.1:47522)
0811 11:55:09.828599 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:09.828647 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:46491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260811 11:55:09.828879 8660 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.124:46491: Network error: Client connection negotiation failed: client connection to 127.3.221.124:46491: connect: Connection refused (error 111)
I20260811 11:55:09.878180 8660 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } attempt: 1
I20260811 11:55:09.879179 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.878825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:46491 (local address 127.0.0.1:47534)
0811 11:55:09.879001 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:09.879078 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:46491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260811 11:55:09.880048 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.879762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:46491 (local address 127.0.0.1:47550)
0811 11:55:09.879912 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:09.879974 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:46491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260811 11:55:09.880230 8660 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.124:46491: Network error: Client connection negotiation failed: client connection to 127.3.221.124:46491: connect: Connection refused (error 111)
I20260811 11:55:09.905982 8530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } attempt: 4
Aug 11 11:55:09 dist-test-slave-bndk krb5kdc[8393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for kudu/127.3.221.125@KRBTEST.COM
I20260811 11:55:09.917143 8669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.906730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.3.221.125:36523)
0811 11:55:09.906879 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:09.906884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:09.906973 (+ 89us) server_negotiation.cc:407] Connection header received
0811 11:55:09.907114 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:09.907117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:09.907166 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:09.907237 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:09.907722 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:09.908275 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:09.909035 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:09.909324 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:09.909422 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:09.911888 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:09.911891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:09.911894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:09.911923 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:09.913852 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:09.914887 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:09.914893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:09.914895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:09.915041 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:09.915477 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:09.915480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:09.915482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:09.915979 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:09.916129 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:09.916634 (+ 505us) server_negotiation.cc:299] Negotiation successful
0811 11:55:09.916767 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 11:55:09.917143 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.906610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:36523 (local address 127.0.0.1:42764)
0811 11:55:09.906889 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:09.906904 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 11:55:09.907017 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:55:09.907277 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:55:09.907280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:55:09.907493 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:55:09.907595 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:09.907601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:09.908385 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:09.908388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:09.908917 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:09.908926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:09.909447 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:09.909451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:09.909558 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:09.910146 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:55:09.910158 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:55:09.911659 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:55:09.914147 (+ 2488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:09.914151 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:09.914154 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:09.914469 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:09.915226 (+ 757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:09.915229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:09.915232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:09.915366 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:09.916174 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:55:09.916178 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:55:09.916310 (+ 132us) client_negotiation.cc:769] Sending connection context
0811 11:55:09.916547 (+ 237us) client_negotiation.cc:240] Negotiation successful
0811 11:55:09.916753 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208}
I20260811 11:55:09.918525 8530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:55:09.919329 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.919028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:46491 (local address 127.0.0.1:47554)
0811 11:55:09.919155 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:09.919214 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:46491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260811 11:55:09.920506 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.920246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:46491 (local address 127.0.0.1:47562)
0811 11:55:09.920384 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:09.920442 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:46491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260811 11:55:09.920692 8530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.124:46491: Network error: Client connection negotiation failed: client connection to 127.3.221.124:46491: connect: Connection refused (error 111)
I20260811 11:55:09.948390 8660 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } attempt: 2
I20260811 11:55:09.949492 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.948972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:46491 (local address 127.0.0.1:47564)
0811 11:55:09.949180 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:09.949293 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:46491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260811 11:55:09.950333 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.950076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:46491 (local address 127.0.0.1:47568)
0811 11:55:09.950221 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:09.950275 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:46491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260811 11:55:09.950544 8660 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.124:46491: Network error: Client connection negotiation failed: client connection to 127.3.221.124:46491: connect: Connection refused (error 111)
W20260811 11:55:09.967545 8672 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 11:55:09.967891 8672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 11:55:09.967927 8672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 11:55:09.967981 8672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 11:55:09.973054 8672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 11:55:09.973165 8672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 11:55:09.973186 8672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 11:55:09.973201 8672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 11:55:09.973254 8672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 11:55:09.979832 8672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.221.84:41777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.221.126:33247,127.3.221.125:36523,127.3.221.124:46491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/krb5kdc/kudu_127.3.221.124.keytab
--principal=kudu/127.3.221.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.221.124:46491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.221.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 11:09:46 UTC on ccycloud.quasar-pllopi.root.comops.site
build id 4580136
I20260811 11:55:09.979907 8530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } attempt: 1
I20260811 11:55:09.980886 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.980456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:46491 (local address 127.0.0.1:47580)
0811 11:55:09.980664 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:09.980735 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:46491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260811 11:55:09.981323 8672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 11:55:09.981716 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:09.981494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:46491 (local address 127.0.0.1:47590)
0811 11:55:09.981600 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:09.981655 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:46491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260811 11:55:09.981868 8530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.124:46491: Network error: Client connection negotiation failed: client connection to 127.3.221.124:46491: connect: Connection refused (error 111)
I20260811 11:55:09.982527 8672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 11:55:09.991928 8682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:55:09.991931 8679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 11:55:09.991933 8680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 11:55:09.992900 8672 server_base.cc:1034] running on GCE node
I20260811 11:55:09.993772 8672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 11:55:09.995091 8672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 11:55:09.996477 8672 hybrid_clock.cc:648] HybridClock initialized: now 1786449309996431 us; error 51 us; skew 500 ppm
Aug 11 11:55:09 dist-test-slave-bndk krb5kdc[8393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 11:55:10.000665 8672 init.cc:377] Logged in from keytab as kudu/127.3.221.124@KRBTEST.COM (short username kudu)
I20260811 11:55:10.001859 8672 webserver.cc:460] Webserver started at http://127.3.221.124:44617/ using document root <none> and password file <none>
I20260811 11:55:10.002503 8672 fs_manager.cc:359] Metadata directory not provided
I20260811 11:55:10.002583 8672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 11:55:10.002818 8672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 11:55:10.005080 8672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-2/data/instance:
uuid: "536f183cd59644329ece425e3b46a8a2"
format_stamp: "Formatted at 2026-08-11 11:55:09 on dist-test-slave-bndk"
server_key: "8343155695b3a8547be9e998018fe6cf"
server_key_iv: "abf0cabe4780b63dc9bc7b14c24d106c"
server_key_version: "encryptionkey@0"
I20260811 11:55:10.005952 8672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-2/wal/instance:
uuid: "536f183cd59644329ece425e3b46a8a2"
format_stamp: "Formatted at 2026-08-11 11:55:09 on dist-test-slave-bndk"
server_key: "8343155695b3a8547be9e998018fe6cf"
server_key_iv: "abf0cabe4780b63dc9bc7b14c24d106c"
server_key_version: "encryptionkey@0"
I20260811 11:55:10.010460 8672 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 11:55:10.013526 8693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 11:55:10.015069 8672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 11:55:10.015223 8672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-2/data,/tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-2/wal
uuid: "536f183cd59644329ece425e3b46a8a2"
format_stamp: "Formatted at 2026-08-11 11:55:09 on dist-test-slave-bndk"
server_key: "8343155695b3a8547be9e998018fe6cf"
server_key_iv: "abf0cabe4780b63dc9bc7b14c24d106c"
server_key_version: "encryptionkey@0"
I20260811 11:55:10.015353 8672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 11:55:10.032694 8672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 11:55:10.036087 8672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 11:55:10.036298 8672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 11:55:10.041713 8660 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } attempt: 3
I20260811 11:55:10.042650 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:10.042328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:46491 (local address 127.0.0.1:47598)
0811 11:55:10.042484 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:10.042552 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:46491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260811 11:55:10.043453 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:10.043184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:46491 (local address 127.0.0.1:47600)
0811 11:55:10.043329 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:10.043382 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.221.124:46491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260811 11:55:10.043605 8660 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.221.124:46491: Network error: Client connection negotiation failed: client connection to 127.3.221.124:46491: connect: Connection refused (error 111)
I20260811 11:55:10.044301 8672 rpc_server.cc:307] RPC server started. Bound to: 127.3.221.124:46491
I20260811 11:55:10.044637 8780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.221.124:46491 every 8 connection(s)
I20260811 11:55:10.045394 8672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-2/data/info.pb
I20260811 11:55:10.046715 3957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt5cui_/build/debug/bin/kudu as pid 8672
I20260811 11:55:10.046836 3957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt5cui_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786449004114841-3957-0/minicluster-data/master-2/wal/instance
I20260811 11:55:10.047132 3957 external_mini_cluster.cc:1428] Setting key a9693f7cbf99827e51c3c3b22ba5cce5
I20260811 11:55:10.048866 8781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 11:55:10.049011 8530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } attempt: 2
I20260811 11:55:10.054513 8781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33247 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:55:10 dist-test-slave-bndk krb5kdc[8393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.126@KRBTEST.COM for kudu/127.3.221.124@KRBTEST.COM
Aug 11 11:55:10 dist-test-slave-bndk krb5kdc[8393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.124@KRBTEST.COM
Aug 11 11:55:10 dist-test-slave-bndk krb5kdc[8393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.124@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:55:10.064539 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:10.049500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:46491 (local address 127.0.0.1:47614)
0811 11:55:10.049638 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:10.049654 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:55:10.049746 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:55:10.051116 (+ 1370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:55:10.051120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:55:10.051362 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:55:10.051480 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:10.051489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.053757 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.053761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:10.054310 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:10.054321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.055181 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.055185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:10.055278 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:10.055839 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:55:10.055853 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:55:10.058840 (+ 2987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:55:10.061931 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:10.061935 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:10.061937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:10.062261 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:10.062766 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:10.062769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:10.062771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:10.062870 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:10.063839 (+ 969us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:55:10.063842 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:55:10.063964 (+ 122us) client_negotiation.cc:769] Sending connection context
0811 11:55:10.064163 (+ 199us) client_negotiation.cc:240] Negotiation successful
0811 11:55:10.064358 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260811 11:55:10.065809 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:10.048683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47604 (local address 127.3.221.124:46491)
0811 11:55:10.049167 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.049191 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.049275 (+ 84us) server_negotiation.cc:407] Connection header received
0811 11:55:10.050228 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.050254 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.050749 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.051146 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.052092 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.053575 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.054804 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.055121 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.055620 (+ 499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:10.059453 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:10.059458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:10.059460 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:10.059506 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:10.061955 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:10.062823 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:10.062828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:10.062830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:10.062940 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:10.063280 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:10.063283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:10.063285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:10.063715 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:10.063910 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:10.064699 (+ 789us) server_negotiation.cc:299] Negotiation successful
0811 11:55:10.064824 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":262,"server-negotiator.queue_time_us":324,"spinlock_wait_cycles":22784,"thread_start_us":151,"threads_started":1}
I20260811 11:55:10.065815 8789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:10.049724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47614 (local address 127.3.221.124:46491)
0811 11:55:10.050011 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.050019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.050033 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:55:10.050231 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.050253 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.050749 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.051670 (+ 921us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.051818 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.053575 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.054439 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.055043 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.055217 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:10.059181 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:10.059188 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:10.059201 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:10.059237 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:10.061773 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:10.062435 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:10.062442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:10.062450 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:10.062634 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:10.063099 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:10.063102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:10.063103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:10.063713 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:10.063852 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:10.064276 (+ 424us) server_negotiation.cc:299] Negotiation successful
0811 11:55:10.064647 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":217,"thread_start_us":133,"threads_started":1}
I20260811 11:55:10.070277 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:10.057344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54020 (local address 127.3.221.126:33247)
0811 11:55:10.057492 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.057496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.057690 (+ 194us) server_negotiation.cc:407] Connection header received
0811 11:55:10.057831 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.057834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.057884 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.057972 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.058653 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.059265 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.060603 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.061000 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.061165 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:10.064841 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:10.064846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:10.064849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:10.064875 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:10.068202 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:10.068819 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:10.068824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:10.068826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:10.068923 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:10.069278 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:10.069281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:10.069282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:10.069519 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:10.069583 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:10.069998 (+ 415us) server_negotiation.cc:299] Negotiation successful
0811 11:55:10.070115 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 11:55:10.070302 8790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:10.057088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.126:33247 (local address 127.0.0.1:54020)
0811 11:55:10.057547 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:10.057604 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 11:55:10.057740 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:55:10.058027 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:55:10.058032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:55:10.058379 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:55:10.058530 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:10.058539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.059430 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.059434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:10.060417 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:10.060433 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.061189 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.061193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:10.061351 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:10.062579 (+ 1228us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:55:10.062594 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:55:10.064681 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:55:10.068316 (+ 3635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:10.068326 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:10.068332 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:10.068691 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:10.069045 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:10.069048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:10.069050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:10.069151 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:10.069655 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:55:10.069660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:55:10.069769 (+ 109us) client_negotiation.cc:769] Sending connection context
0811 11:55:10.069884 (+ 115us) client_negotiation.cc:240] Negotiation successful
0811 11:55:10.070106 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":171,"threads_started":1}
I20260811 11:55:10.072160 8530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d: Bootstrap starting.
I20260811 11:55:10.075973 8781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } has no permanent_uuid. Determining permanent_uuid...
I20260811 11:55:10.077689 8530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d: Neither blocks nor log segments found. Creating new log.
Aug 11 11:55:10 dist-test-slave-bndk krb5kdc[8393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.126@KRBTEST.COM
I20260811 11:55:10.078867 8530 log.cc:824] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d: Log is configured to *not* fsync() on all Append() calls
I20260811 11:55:10.082383 8530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d: No bootstrap required, opened a new log
I20260811 11:55:10.082895 8791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:10.068614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54032 (local address 127.3.221.126:33247)
0811 11:55:10.068975 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.068989 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.069001 (+ 12us) server_negotiation.cc:407] Connection header received
0811 11:55:10.069036 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.069038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.069082 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.069173 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.069989 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.071953 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.073052 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.073474 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.074745 (+ 1271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:10.079144 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:10.079149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:10.079152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:10.079187 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:10.080779 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:10.081383 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:10.081387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:10.081389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:10.081502 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:10.081835 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:10.081838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:10.081840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:10.082088 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:10.082273 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:10.082521 (+ 248us) server_negotiation.cc:299] Negotiation successful
0811 11:55:10.082673 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":113,"threads_started":1}
Aug 11 11:55:10 dist-test-slave-bndk krb5kdc[8393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.124@KRBTEST.COM for kudu/127.3.221.125@KRBTEST.COM
I20260811 11:55:10.087255 8530 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c60b746f6c4d199836aac0f09c357d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33247 } } peers { permanent_uuid: "8bfdc03be7484a3db1490e429309cb11" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } } peers { permanent_uuid: "536f183cd59644329ece425e3b46a8a2" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } }
I20260811 11:55:10.087554 8530 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c60b746f6c4d199836aac0f09c357d, State: Initialized, Role: FOLLOWER
I20260811 11:55:10.087942 8530 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c60b746f6c4d199836aac0f09c357d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33247 } } peers { permanent_uuid: "8bfdc03be7484a3db1490e429309cb11" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } } peers { permanent_uuid: "536f183cd59644329ece425e3b46a8a2" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } }
I20260811 11:55:10.088575 8795 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c60b746f6c4d199836aac0f09c357d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33247 } } peers { permanent_uuid: "8bfdc03be7484a3db1490e429309cb11" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } } peers { permanent_uuid: "536f183cd59644329ece425e3b46a8a2" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } } }
I20260811 11:55:10.088874 8669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:10.078151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42776 (local address 127.3.221.125:36523)
0811 11:55:10.078330 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.078335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.078865 (+ 530us) server_negotiation.cc:407] Connection header received
0811 11:55:10.079130 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.079134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.079218 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.079270 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.080658 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.081334 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.082356 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.082669 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.082792 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:10.085444 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:10.085448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:10.085451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:10.085475 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:10.087121 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:10.087538 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:10.087542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:10.087544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:10.087632 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:10.087914 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:10.087918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:10.087920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:10.088115 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:10.088204 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:10.088573 (+ 369us) server_negotiation.cc:299] Negotiation successful
0811 11:55:10.088709 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 11:55:10.088868 8795 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:55:10.089437 8790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:10.078057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.125:36523 (local address 127.0.0.1:42776)
0811 11:55:10.078471 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:10.078488 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 11:55:10.078955 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:55:10.079476 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:55:10.079479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:55:10.079904 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:55:10.080397 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:10.080411 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.081462 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.081467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:10.082173 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:10.082186 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.083135 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.083144 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:10.083267 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:10.083794 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:55:10.083809 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:55:10.085277 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:55:10.087222 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:10.087226 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:10.087228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:10.087443 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:10.087738 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:10.087741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:10.087743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:10.087824 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:10.088250 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:55:10.088254 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:55:10.088360 (+ 106us) client_negotiation.cc:769] Sending connection context
0811 11:55:10.089095 (+ 735us) client_negotiation.cc:240] Negotiation successful
0811 11:55:10.089304 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319}
I20260811 11:55:10.090036 8530 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:55:10.090248 8781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 11:55:10 dist-test-slave-bndk krb5kdc[8393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.221.125@KRBTEST.COM
Aug 11 11:55:10 dist-test-slave-bndk krb5kdc[8393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.124@KRBTEST.COM for kudu/127.3.221.124@KRBTEST.COM
I20260811 11:55:10.101600 8794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:10.084527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42784 (local address 127.3.221.125:36523)
0811 11:55:10.085045 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.085055 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.085070 (+ 15us) server_negotiation.cc:407] Connection header received
0811 11:55:10.085114 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.085117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.085173 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.085312 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.086053 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.086879 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.087929 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.088281 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.088383 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:10.092798 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:10.092802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:10.092805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:10.092838 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:10.097767 (+ 4929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:10.098713 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:10.098728 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:10.098730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:10.098854 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:10.099853 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:10.099857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:10.099859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:10.100183 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:10.100387 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:10.101202 (+ 815us) server_negotiation.cc:299] Negotiation successful
0811 11:55:10.101389 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":123,"threads_started":1}
W20260811 11:55:10.101907 8815 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:55:10.102003 8815 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:55:10.100996 8414 master_runner.cc:186] Error getting master registration for 127.3.221.125:36523: OK, instance_id { permanent_uuid: "8bfdc03be7484a3db1490e429309cb11" instance_seqno: 1786449309776015 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 11:55:10.103999 8414 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 11:55:10.105499 8790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:10.091442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:46491 (local address 127.0.0.1:47622)
0811 11:55:10.091892 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:10.091908 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 11:55:10.092012 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:55:10.093948 (+ 1936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:55:10.093952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:55:10.094134 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:55:10.094247 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:10.094253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.095796 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.095801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:10.096964 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:10.096980 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.098991 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.099003 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:10.099103 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:10.099801 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:55:10.099825 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:55:10.101785 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:55:10.103773 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:10.103776 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:10.103777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:10.104023 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:10.104297 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:10.104300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:10.104302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:10.104389 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:10.104915 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:55:10.104918 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:55:10.105017 (+ 99us) client_negotiation.cc:769] Sending connection context
0811 11:55:10.105129 (+ 112us) client_negotiation.cc:240] Negotiation successful
0811 11:55:10.105324 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365}
I20260811 11:55:10.105612 8789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:10.091850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47622 (local address 127.3.221.124:46491)
0811 11:55:10.091942 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.091946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.091960 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:55:10.092097 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.092099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.092140 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.092208 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.094516 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.095368 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.097387 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.098307 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.098755 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:10.101944 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:10.101948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:10.101950 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:10.101976 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:10.103642 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:10.104128 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:10.104131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:10.104133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:10.104213 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:10.104490 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:10.104494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:10.104496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:10.104812 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:10.104916 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:10.105210 (+ 294us) server_negotiation.cc:299] Negotiation successful
0811 11:55:10.105411 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":39}
I20260811 11:55:10.108618 8781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 536f183cd59644329ece425e3b46a8a2: Bootstrap starting.
I20260811 11:55:10.110615 3957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 11:55:10.111663 8781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 536f183cd59644329ece425e3b46a8a2: Neither blocks nor log segments found. Creating new log.
I20260811 11:55:10.113592 8482 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d2c60b746f6c4d199836aac0f09c357d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54032
I20260811 11:55:10.114488 8781 log.cc:824] T 00000000000000000000000000000000 P 536f183cd59644329ece425e3b46a8a2: Log is configured to *not* fsync() on all Append() calls
I20260811 11:55:10.114485 8482 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260811 11:55:10.114672 8482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:55:10.116779 8482 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c60b746f6c4d199836aac0f09c357d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33247 } } peers { permanent_uuid: "8bfdc03be7484a3db1490e429309cb11" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } } peers { permanent_uuid: "536f183cd59644329ece425e3b46a8a2" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } }
I20260811 11:55:10.118523 8781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 536f183cd59644329ece425e3b46a8a2: No bootstrap required, opened a new log
I20260811 11:55:10.120010 8482 leader_election.cc:288] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d [CANDIDATE]: Term 1 election: Requested vote from peers 8bfdc03be7484a3db1490e429309cb11 (127.3.221.125:36523), 536f183cd59644329ece425e3b46a8a2 (127.3.221.124:46491)
I20260811 11:55:10.122815 8781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 536f183cd59644329ece425e3b46a8a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c60b746f6c4d199836aac0f09c357d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33247 } } peers { permanent_uuid: "8bfdc03be7484a3db1490e429309cb11" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } } peers { permanent_uuid: "536f183cd59644329ece425e3b46a8a2" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } }
I20260811 11:55:10.123164 8781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 536f183cd59644329ece425e3b46a8a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 536f183cd59644329ece425e3b46a8a2, State: Initialized, Role: FOLLOWER
I20260811 11:55:10.123569 8781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 536f183cd59644329ece425e3b46a8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c60b746f6c4d199836aac0f09c357d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33247 } } peers { permanent_uuid: "8bfdc03be7484a3db1490e429309cb11" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } } peers { permanent_uuid: "536f183cd59644329ece425e3b46a8a2" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } }
I20260811 11:55:10.125298 8781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 536f183cd59644329ece425e3b46a8a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:55:10.125404 8735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2c60b746f6c4d199836aac0f09c357d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "536f183cd59644329ece425e3b46a8a2"
I20260811 11:55:10.125821 8735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 536f183cd59644329ece425e3b46a8a2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:55:10.127574 8735 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 536f183cd59644329ece425e3b46a8a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2c60b746f6c4d199836aac0f09c357d in term 1.
I20260811 11:55:10.128145 8443 leader_election.cc:302] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 536f183cd59644329ece425e3b46a8a2, d2c60b746f6c4d199836aac0f09c357d; no voters:
I20260811 11:55:10.128535 8795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 11:55:10.128738 8795 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d [term 1 LEADER]: Becoming Leader. State: Replica: d2c60b746f6c4d199836aac0f09c357d, State: Running, Role: LEADER
I20260811 11:55:10.129146 8795 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c60b746f6c4d199836aac0f09c357d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33247 } } peers { permanent_uuid: "8bfdc03be7484a3db1490e429309cb11" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } } peers { permanent_uuid: "536f183cd59644329ece425e3b46a8a2" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } }
W20260811 11:55:10.133697 8848 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 536f183cd59644329ece425e3b46a8a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:55:10.133808 8848 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:55:10.134482 8834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 536f183cd59644329ece425e3b46a8a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c60b746f6c4d199836aac0f09c357d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33247 } } peers { permanent_uuid: "8bfdc03be7484a3db1490e429309cb11" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } } peers { permanent_uuid: "536f183cd59644329ece425e3b46a8a2" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } } }
I20260811 11:55:10.134727 8834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 536f183cd59644329ece425e3b46a8a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:55:10.136258 8791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:10.123671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54038 (local address 127.3.221.126:33247)
0811 11:55:10.123791 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.123795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.124125 (+ 330us) server_negotiation.cc:407] Connection header received
0811 11:55:10.124315 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.124319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.124371 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.124425 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.125562 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.126356 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.128307 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.128806 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.128907 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:10.131873 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:10.131877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:10.131880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:10.131912 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:10.134147 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:10.134692 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:10.134696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:10.134698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:10.134834 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:10.135175 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:10.135179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:10.135181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:10.135441 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:10.135548 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:10.135946 (+ 398us) server_negotiation.cc:299] Negotiation successful
0811 11:55:10.136090 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 11:55:10.138659 8795 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2c60b746f6c4d199836aac0f09c357d. Latest consensus state: current_term: 1 leader_uuid: "d2c60b746f6c4d199836aac0f09c357d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c60b746f6c4d199836aac0f09c357d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33247 } } peers { permanent_uuid: "8bfdc03be7484a3db1490e429309cb11" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } } peers { permanent_uuid: "536f183cd59644329ece425e3b46a8a2" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } } }
I20260811 11:55:10.138906 8795 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d [sys.catalog]: This master's current role is: LEADER
I20260811 11:55:10.139124 8672 master_runner.cc:186] Error getting master registration for 127.3.221.125:36523: OK, instance_id { permanent_uuid: "8bfdc03be7484a3db1490e429309cb11" instance_seqno: 1786449309776015 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 11:55:10.142838 8672 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 11:55:10.147346 8853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 11:55:10.147377 8794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:10.123289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42792 (local address 127.3.221.125:36523)
0811 11:55:10.123469 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.123473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.138007 (+ 14534us) server_negotiation.cc:407] Connection header received
0811 11:55:10.138198 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.138202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.138266 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.138325 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.139243 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.140037 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.141162 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.141568 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.141651 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:10.143497 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:10.143501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:10.143504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:10.143538 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:10.145325 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:10.145866 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:10.145870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:10.145872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:10.145965 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:10.146278 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:10.146281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:10.146283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:10.146519 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:10.146627 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:10.147061 (+ 434us) server_negotiation.cc:299] Negotiation successful
0811 11:55:10.147203 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260811 11:55:10.148804 8853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 11:55:10.152544 8789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:10.124009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47624 (local address 127.3.221.124:46491)
0811 11:55:10.124156 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.124160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.139653 (+ 15493us) server_negotiation.cc:407] Connection header received
0811 11:55:10.139900 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.139904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.139969 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.140019 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.141660 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.142374 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.144051 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.144413 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.144487 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:10.148455 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:10.148460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:10.148463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:10.148494 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:10.150248 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:10.151003 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:10.151007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:10.151009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:10.151117 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:10.151486 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:10.151490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:10.151492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:10.151742 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:10.151892 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:10.152244 (+ 352us) server_negotiation.cc:299] Negotiation successful
0811 11:55:10.152377 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 11:55:10.157498 8735 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 536f183cd59644329ece425e3b46a8a2 [term 1 FOLLOWER]: Refusing update from remote peer d2c60b746f6c4d199836aac0f09c357d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:55:10.158509 8849 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d [LEADER]: Connected to new peer: Peer: permanent_uuid: "536f183cd59644329ece425e3b46a8a2" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:55:10.164497 8834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 536f183cd59644329ece425e3b46a8a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2c60b746f6c4d199836aac0f09c357d. Latest consensus state: current_term: 1 leader_uuid: "d2c60b746f6c4d199836aac0f09c357d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c60b746f6c4d199836aac0f09c357d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33247 } } peers { permanent_uuid: "8bfdc03be7484a3db1490e429309cb11" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } } peers { permanent_uuid: "536f183cd59644329ece425e3b46a8a2" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } } }
I20260811 11:55:10.164690 8834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 536f183cd59644329ece425e3b46a8a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:55:10.169724 8849 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d2c60b746f6c4d199836aac0f09c357d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c60b746f6c4d199836aac0f09c357d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33247 } } peers { permanent_uuid: "8bfdc03be7484a3db1490e429309cb11" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } } peers { permanent_uuid: "536f183cd59644329ece425e3b46a8a2" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } } }
I20260811 11:55:10.169925 8849 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d [sys.catalog]: This master's current role is: LEADER
I20260811 11:55:10.170060 8854 mvcc.cc:205] Tried to move back new op lower bound from 7317296374394179584 to 7317296374295433216. Current Snapshot: MvccSnapshot[applied={T|T < 7317296374394179584}]
I20260811 11:55:10.172365 8834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 536f183cd59644329ece425e3b46a8a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d2c60b746f6c4d199836aac0f09c357d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c60b746f6c4d199836aac0f09c357d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33247 } } peers { permanent_uuid: "8bfdc03be7484a3db1490e429309cb11" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } } peers { permanent_uuid: "536f183cd59644329ece425e3b46a8a2" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } } }
I20260811 11:55:10.172708 8834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 536f183cd59644329ece425e3b46a8a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:55:10.174444 8855 mvcc.cc:205] Tried to move back new op lower bound from 7317296374394179584 to 7317296374295433216. Current Snapshot: MvccSnapshot[applied={T|T < 7317296374394179584}]
I20260811 11:55:10.177664 8853 catalog_manager.cc:1347] Generated new cluster ID: 06b4aec8c31a46bd87556e24a49fbbe0
I20260811 11:55:10.177785 8853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 11:55:10.186767 8660 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } attempt: 4
Aug 11 11:55:10 dist-test-slave-bndk krb5kdc[8393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786449309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.221.125@KRBTEST.COM for kudu/127.3.221.124@KRBTEST.COM
I20260811 11:55:10.201996 8789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:10.188233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47628 (local address 127.3.221.124:46491)
0811 11:55:10.188361 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.188365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.188379 (+ 14us) server_negotiation.cc:407] Connection header received
0811 11:55:10.188427 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.188430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.188478 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.188552 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.189183 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.190107 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.191633 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.192001 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.192107 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:10.195573 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 11:55:10.195578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 11:55:10.195581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 11:55:10.195615 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 11:55:10.197606 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:10.198883 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:10.198889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:10.198891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:10.199063 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 11:55:10.199955 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 11:55:10.199961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 11:55:10.199963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 11:55:10.200529 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 11:55:10.200684 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:10.201358 (+ 674us) server_negotiation.cc:299] Negotiation successful
0811 11:55:10.201674 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 11:55:10.203507 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:10.187499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.221.124:46491 (local address 127.0.0.1:47628)
0811 11:55:10.187939 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0811 11:55:10.187956 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 11:55:10.188064 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 11:55:10.188685 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 11:55:10.188688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 11:55:10.188922 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0811 11:55:10.189041 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:10.189048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.190329 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.190333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:10.191211 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 11:55:10.191373 (+ 162us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 11:55:10.192317 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 11:55:10.192322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 11:55:10.192423 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:10.193251 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 11:55:10.193277 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 11:55:10.195323 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 11:55:10.197815 (+ 2492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:10.197819 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:10.197822 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:10.198519 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:10.199382 (+ 863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 11:55:10.199386 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 11:55:10.199389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 11:55:10.199717 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 11:55:10.200819 (+ 1102us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 11:55:10.200823 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 11:55:10.201018 (+ 195us) client_negotiation.cc:769] Sending connection context
0811 11:55:10.203124 (+ 2106us) client_negotiation.cc:240] Negotiation successful
0811 11:55:10.203331 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341}
I20260811 11:55:10.207088 8660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bfdc03be7484a3db1490e429309cb11: Bootstrap starting.
I20260811 11:55:10.210569 8660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bfdc03be7484a3db1490e429309cb11: Neither blocks nor log segments found. Creating new log.
I20260811 11:55:10.212096 8660 log.cc:824] T 00000000000000000000000000000000 P 8bfdc03be7484a3db1490e429309cb11: Log is configured to *not* fsync() on all Append() calls
I20260811 11:55:10.212917 8853 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 11:55:10.215173 8853 catalog_manager.cc:1504] Loading token signing keys...
I20260811 11:55:10.217979 8660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bfdc03be7484a3db1490e429309cb11: No bootstrap required, opened a new log
I20260811 11:55:10.226971 8660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bfdc03be7484a3db1490e429309cb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c60b746f6c4d199836aac0f09c357d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33247 } } peers { permanent_uuid: "8bfdc03be7484a3db1490e429309cb11" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } } peers { permanent_uuid: "536f183cd59644329ece425e3b46a8a2" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } }
I20260811 11:55:10.227896 8660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bfdc03be7484a3db1490e429309cb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bfdc03be7484a3db1490e429309cb11, State: Initialized, Role: FOLLOWER
I20260811 11:55:10.228688 8660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bfdc03be7484a3db1490e429309cb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c60b746f6c4d199836aac0f09c357d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33247 } } peers { permanent_uuid: "8bfdc03be7484a3db1490e429309cb11" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } } peers { permanent_uuid: "536f183cd59644329ece425e3b46a8a2" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } }
I20260811 11:55:10.232131 8611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2c60b746f6c4d199836aac0f09c357d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8bfdc03be7484a3db1490e429309cb11"
I20260811 11:55:10.232705 8611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bfdc03be7484a3db1490e429309cb11 [term 0 FOLLOWER]: Advancing to term 1
I20260811 11:55:10.234995 8611 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8bfdc03be7484a3db1490e429309cb11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2c60b746f6c4d199836aac0f09c357d in term 1.
I20260811 11:55:10.236560 8608 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8bfdc03be7484a3db1490e429309cb11 [term 1 FOLLOWER]: Refusing update from remote peer d2c60b746f6c4d199836aac0f09c357d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 11:55:10.237169 8660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bfdc03be7484a3db1490e429309cb11 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 11:55:10.239557 8849 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bfdc03be7484a3db1490e429309cb11" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 11:55:10.240969 8863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bfdc03be7484a3db1490e429309cb11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c60b746f6c4d199836aac0f09c357d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33247 } } peers { permanent_uuid: "8bfdc03be7484a3db1490e429309cb11" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } } peers { permanent_uuid: "536f183cd59644329ece425e3b46a8a2" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } } }
I20260811 11:55:10.241169 8863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bfdc03be7484a3db1490e429309cb11 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:55:10.245152 8853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d: Generated new TSK 0
I20260811 11:55:10.246107 8853 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 11:55:10.246376 8884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bfdc03be7484a3db1490e429309cb11 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d2c60b746f6c4d199836aac0f09c357d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c60b746f6c4d199836aac0f09c357d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33247 } } peers { permanent_uuid: "8bfdc03be7484a3db1490e429309cb11" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } } peers { permanent_uuid: "536f183cd59644329ece425e3b46a8a2" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } } }
I20260811 11:55:10.246526 8884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bfdc03be7484a3db1490e429309cb11 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:55:10.247300 8863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bfdc03be7484a3db1490e429309cb11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2c60b746f6c4d199836aac0f09c357d. Latest consensus state: current_term: 1 leader_uuid: "d2c60b746f6c4d199836aac0f09c357d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c60b746f6c4d199836aac0f09c357d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33247 } } peers { permanent_uuid: "8bfdc03be7484a3db1490e429309cb11" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } } peers { permanent_uuid: "536f183cd59644329ece425e3b46a8a2" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } } }
I20260811 11:55:10.247416 8863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bfdc03be7484a3db1490e429309cb11 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 11:55:10.248747 8795 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d2c60b746f6c4d199836aac0f09c357d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c60b746f6c4d199836aac0f09c357d" member_type: VOTER last_known_addr { host: "127.3.221.126" port: 33247 } } peers { permanent_uuid: "8bfdc03be7484a3db1490e429309cb11" member_type: VOTER last_known_addr { host: "127.3.221.125" port: 36523 } } peers { permanent_uuid: "536f183cd59644329ece425e3b46a8a2" member_type: VOTER last_known_addr { host: "127.3.221.124" port: 46491 } } }
I20260811 11:55:10.248883 8795 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2c60b746f6c4d199836aac0f09c357d [sys.catalog]: This master's current role is: LEADER
W20260811 11:55:10.254436 8822 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.ConnectToMaster -> {remote=127.3.221.126:33247, user_credentials={real_user=slave}} blocked reactor thread for 97669.7us
W20260811 11:55:10.254488 8826 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.ConnectToMaster -> {remote=127.3.221.125:36523, user_credentials={real_user=slave}} blocked reactor thread for 95764.8us
W20260811 11:55:10.258173 8878 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8bfdc03be7484a3db1490e429309cb11: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 11:55:10.258285 8878 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 11:55:10.261248 8881 mvcc.cc:205] Tried to move back new op lower bound from 7317296374714691584 to 7317296374295433216. Current Snapshot: MvccSnapshot[applied={T|T < 7317296374394179584}]
W20260811 11:55:10.301693 8789 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:10.297886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47644 (local address 127.3.221.124:46491)
0811 11:55:10.298037 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.298041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.300198 (+ 2157us) server_negotiation.cc:407] Connection header received
0811 11:55:10.300600 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.300605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.300665 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.300727 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.301535 (+ 808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47644: BlockingRecv error: recv got EOF from 127.0.0.1:47644 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260811 11:55:10.303422 8791 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:10.299209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54048 (local address 127.3.221.126:33247)
0811 11:55:10.299348 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.299352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.302320 (+ 2968us) server_negotiation.cc:407] Connection header received
0811 11:55:10.302529 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.302532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.302592 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.302649 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.303275 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54048: BlockingRecv error: recv got EOF from 127.0.0.1:54048 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260811 11:55:10.305397 8794 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:10.298986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42800 (local address 127.3.221.125:36523)
0811 11:55:10.299137 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.299142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.304231 (+ 5089us) server_negotiation.cc:407] Connection header received
0811 11:55:10.304455 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.304458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.304522 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.304581 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.305210 (+ 629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42800: BlockingRecv error: recv got EOF from 127.0.0.1:42800 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260811 11:55:10.311836 8794 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:10.309355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42808 (local address 127.3.221.125:36523)
0811 11:55:10.309500 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.309505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.310178 (+ 673us) server_negotiation.cc:407] Connection header received
0811 11:55:10.310503 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.310508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.310609 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.310676 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.311698 (+ 1022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42808: BlockingRecv error: recv got EOF from 127.0.0.1:42808 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260811 11:55:10.319630 8794 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:10.317923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42816 (local address 127.3.221.125:36523)
0811 11:55:10.318090 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.318095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.318473 (+ 378us) server_negotiation.cc:407] Connection header received
0811 11:55:10.318719 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.318723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.318788 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.318848 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.319496 (+ 648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42816: BlockingRecv error: recv got EOF from 127.0.0.1:42816 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260811 11:55:10.327263 8794 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:10.324980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42822 (local address 127.3.221.125:36523)
0811 11:55:10.325121 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.325125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.325852 (+ 727us) server_negotiation.cc:407] Connection header received
0811 11:55:10.326078 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.326081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.326141 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.326195 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.327135 (+ 940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42822: BlockingRecv error: recv got EOF from 127.0.0.1:42822 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260811 11:55:10.336227 8794 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:10.334374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42824 (local address 127.3.221.125:36523)
0811 11:55:10.334512 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.334517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.335135 (+ 618us) server_negotiation.cc:407] Connection header received
0811 11:55:10.335360 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.335364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.335423 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.335472 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.336083 (+ 611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42824: BlockingRecv error: recv got EOF from 127.0.0.1:42824 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260811 11:55:10.348510 8794 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:10.347223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42834 (local address 127.3.221.125:36523)
0811 11:55:10.347352 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.347357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.347624 (+ 267us) server_negotiation.cc:407] Connection header received
0811 11:55:10.347754 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.347757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.347821 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.347918 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.348381 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42834: BlockingRecv error: recv got EOF from 127.0.0.1:42834 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260811 11:55:10.369526 8794 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:10.367614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.3.221.125:36523)
0811 11:55:10.367882 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.367888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.368213 (+ 325us) server_negotiation.cc:407] Connection header received
0811 11:55:10.368346 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.368350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.368523 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.368751 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.369393 (+ 642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42840: BlockingRecv error: recv got EOF from 127.0.0.1:42840 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
W20260811 11:55:10.405853 8794 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:10.403940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42852 (local address 127.3.221.125:36523)
0811 11:55:10.404110 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.404115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.404646 (+ 531us) server_negotiation.cc:407] Connection header received
0811 11:55:10.404832 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.404836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.404910 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.405041 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.405716 (+ 675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42852: BlockingRecv error: recv got EOF from 127.0.0.1:42852 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260811 11:55:10.473475 8794 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:10.472310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42858 (local address 127.3.221.125:36523)
0811 11:55:10.472493 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.472497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.472515 (+ 18us) server_negotiation.cc:407] Connection header received
0811 11:55:10.472661 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.472663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.472717 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.472792 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.473364 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42858: BlockingRecv error: recv got EOF from 127.0.0.1:42858 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260811 11:55:10.604810 8794 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:10.603573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42868 (local address 127.3.221.125:36523)
0811 11:55:10.603708 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.603712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.603828 (+ 116us) server_negotiation.cc:407] Connection header received
0811 11:55:10.603916 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.603919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.603979 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.604068 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.604591 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42868: BlockingRecv error: recv got EOF from 127.0.0.1:42868 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260811 11:55:10.864513 8794 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 11:55:10.863514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42882 (local address 127.3.221.125:36523)
0811 11:55:10.863646 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:10.863650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:10.863666 (+ 16us) server_negotiation.cc:407] Connection header received
0811 11:55:10.863751 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:10.863753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:10.863804 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:10.863878 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 11:55:10.864411 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42882: BlockingRecv error: recv got EOF from 127.0.0.1:42882 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 11:55:11.135237 8848 catalog_manager.cc:1259] Loaded cluster ID: 06b4aec8c31a46bd87556e24a49fbbe0
I20260811 11:55:11.135339 8848 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 536f183cd59644329ece425e3b46a8a2: loading cluster ID for follower catalog manager: success
I20260811 11:55:11.137048 8848 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 536f183cd59644329ece425e3b46a8a2: acquiring CA information for follower catalog manager: success
I20260811 11:55:11.138332 8848 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 536f183cd59644329ece425e3b46a8a2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 11:55:11.259660 8878 catalog_manager.cc:1259] Loaded cluster ID: 06b4aec8c31a46bd87556e24a49fbbe0
I20260811 11:55:11.259757 8878 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8bfdc03be7484a3db1490e429309cb11: loading cluster ID for follower catalog manager: success
I20260811 11:55:11.261693 8878 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8bfdc03be7484a3db1490e429309cb11: acquiring CA information for follower catalog manager: success
I20260811 11:55:11.262995 8878 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8bfdc03be7484a3db1490e429309cb11: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 11:55:11.383544 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:11.378820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42896 (local address 127.3.221.125:36523)
0811 11:55:11.379095 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:11.379098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:11.379115 (+ 17us) server_negotiation.cc:407] Connection header received
0811 11:55:11.379169 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:11.379172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:11.379303 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:11.379429 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:55:11.379992 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:11.380690 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:11.381737 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:11.381989 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:11.382085 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:11.382520 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:55:11.383031 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:55:11.383146 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:11.383337 (+ 191us) server_negotiation.cc:299] Negotiation successful
0811 11:55:11.383403 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":95,"threads_started":1}
I20260811 11:55:11.392062 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:11.387364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47658 (local address 127.3.221.124:46491)
0811 11:55:11.387623 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:11.387627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:11.387867 (+ 240us) server_negotiation.cc:407] Connection header received
0811 11:55:11.388033 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:11.388036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:11.388174 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:11.388302 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:55:11.388654 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:11.389207 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:11.390176 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:11.390459 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:11.390557 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:11.390931 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:55:11.391498 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:55:11.391604 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:11.391783 (+ 179us) server_negotiation.cc:299] Negotiation successful
0811 11:55:11.391918 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
I20260811 11:55:11.401746 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:11.397081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47660 (local address 127.3.221.124:46491)
0811 11:55:11.397289 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:11.397295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:11.397814 (+ 519us) server_negotiation.cc:407] Connection header received
0811 11:55:11.398042 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:11.398046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:11.398117 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:11.398187 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:55:11.398456 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:11.399361 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:11.400412 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:11.400864 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:11.400999 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:11.401274 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:55:11.401354 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:55:11.401416 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:11.401566 (+ 150us) server_negotiation.cc:299] Negotiation successful
0811 11:55:11.401629 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260811 11:55:11.401746 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:11.397051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42902 (local address 127.3.221.125:36523)
0811 11:55:11.397297 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:11.397301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:11.397335 (+ 34us) server_negotiation.cc:407] Connection header received
0811 11:55:11.397405 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:11.397408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:11.397469 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:11.397567 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:55:11.397914 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:11.398807 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:11.400164 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:11.400461 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:11.400567 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:11.401163 (+ 596us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:55:11.401301 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:55:11.401392 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:11.401502 (+ 110us) server_negotiation.cc:299] Negotiation successful
0811 11:55:11.401589 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260811 11:55:11.402726 9024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 11:55:11.396685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54050 (local address 127.3.221.126:33247)
0811 11:55:11.397548 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0811 11:55:11.397553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 11:55:11.397936 (+ 383us) server_negotiation.cc:407] Connection header received
0811 11:55:11.398054 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 11:55:11.398058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 11:55:11.398435 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 11:55:11.398620 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 11:55:11.399101 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:11.399962 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:11.400934 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 11:55:11.401349 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 11:55:11.401454 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 11:55:11.401855 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 11:55:11.402249 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 11:55:11.402354 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0811 11:55:11.402501 (+ 147us) server_negotiation.cc:299] Negotiation successful
0811 11:55:11.402571 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":97,"threads_started":1}
I20260811 11:55:11.406047 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 8414
I20260811 11:55:11.435547 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 8540
I20260811 11:55:11.452383 3957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt5cui_/build/debug/bin/kudu with pid 8672
2026-08-11T11:55:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3531 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6971 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (307313 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST